Word error rate, and what it leaves out

Word error rate, usually shortened to WER, is the number most often quoted for a speech recognizer. Computing it takes three steps. Line up the machine’s transcript against a reference transcript typed by a person. Count the words the machine got wrong, the words it missed and the words it added. Divide that total by the number of words in the reference.

In the usual notation, with S for substitutions, D for deletions, I for insertions and N for the words in the reference:

WER = (S + D + I) / N

The alignment is a minimum edit distance, the same idea a spell checker uses to decide how far one string sits from another, applied to words instead of letters. The reference implementation in research is the speech scoring toolkit that the National Institute of Standards and Technology publishes as SCTK, whose sclite program has long been used to score its evaluations.

A worked example

Take a seven-word order: “sell fifteen million cable at thirty two”. The recognizer returns “sell fifty million of cable at thirty two”. Aligned, fifteen became fifty, which is one substitution, and “of” appeared from nowhere, which is one insertion. Two errors over seven reference words gives 28.6 percent.

Both errors count the same. The first cut the size of the order to less than a third of what was asked for, and anyone booking from that transcript would book the wrong trade. The second added a word nobody will ever notice. A reader handed the figure of 28.6 percent has no way to tell which kind of mistake produced it.

Why it can pass 100 percent

Insertions count, and the denominator is only the length of the reference. A system that invents words over silence, hold music or someone else’s conversation on an open line can therefore score above 100 percent. This surprises people who read WER as the share of words that came out wrong. It is a count of edits measured against the reference, and on audio with little real speech in it the count can overtake the total.

Normalization moves the score

Before scoring, both transcripts are usually normalized: case folded, punctuation stripped, numbers and abbreviations written one consistent way. Whether “32” and “thirty two” count as a match depends entirely on that step, and so does “okay” against “OK”. Two published results on the same audio can sit several points apart because one team expanded numerals and the other did not. A figure quoted without its normalization rules is hard to compare with anything else.

The benchmark numbers

The best-known results come from the NIST 2000 conversational telephone test set. In a 2016 paper, a corporate research team measured professional transcribers at 5.9 percent on its Switchboard portion, where strangers discuss an assigned topic, and 11.3 percent on its CallHome portion, where friends and relatives talk freely. Its system scored 5.8 and 11.0 percent. A few months later another lab measured the best human transcriber at 5.1 percent on Switchboard after quality checks.

The gap between the two portions says more than the headline does. Humans and machines alike made nearly twice as many errors once the speakers knew each other and talked the way people really talk.

Measuring what a desk cares about

For market audio, the useful measures sit next to WER rather than in place of it. Some teams score only the tokens that carry a trade (sizes, prices, instrument names and the words for direction), and report an error rate over those alone. Others weight each error by how much a reader would care about it. Neither approach has one agreed formula, and that is the main reason WER survives: everyone computes it the same way, even when it measures the wrong thing.