Accents, crosstalk and the phone line: where speech recognition slips
A recognizer that works well in a quiet office can make nearly twice as many errors on a different group of voices, and more again on a crowded phone line. The reasons are mostly about what the model heard in training.
A speech recognizer is, in the end, a statistical guess about what someone probably said, shaped by the audio it was trained on. When the voice in front of it sounds like the voices in its training set, the guess is good. When it does not, the errors climb, and they climb unevenly. For a trading floor, where one desk can hold speakers from many countries talking over one another on phone lines, that unevenness is the first thing to understand about any transcription system.
What the research found on accents
The clearest measurement so far came from a Stanford-led study published in the Proceedings of the National Academy of Sciences in March 2020. The researchers ran five commercial recognizers over 19.8 hours of structured interviews with 42 white speakers and 73 Black speakers from five U.S. cities, matched on age and gender. Every system did worse on the Black speakers. The average word error rate was 0.35 for them against 0.19 for the white speakers, meaning roughly one word in three came out wrong instead of one in five.
The authors then looked at why. On a subset of identical phrases spoken by both groups, the gap was just as large, so the problem was not vocabulary or grammar. They traced it to the acoustic models, the part of a recognizer that maps sound to speech units, and pointed to training data that underrepresented how many people actually speak.
The same mechanism applies to regional and second-language accents. A model trained mostly on one variety of English learns what that variety’s vowels and rhythms sound like. A trader from Glasgow, Lagos or Mumbai shifts those patterns, and the model has less evidence to go on. The effect varies by system and by speaker, and nobody should assume a figure measured on one population transfers to another.
The phone line takes something away
Much market audio still travels over telephony, and classic telephone channels carry speech in a narrow band, roughly 300 to 3,400 hertz, sampled at 8,000 times a second. That band keeps speech intelligible to a human listener who can lean on context. It also discards most of the energy that distinguishes “s” from “f” and “th”, which sits higher up. For a recognizer, “fifteen” and “fifty” start to look alike, and on a trading call those two words are not interchangeable.
Compression adds its own damage. Voice over IP codecs, mobile networks and recording systems that store audio at low bit rates each smear the signal a little further, and the losses stack.
Crosstalk and open lines
A trading desk adds problems that a call center rarely has. Turrets run several lines at once, some of them open speaker channels that play continuously. A handset microphone picks up the trader next to the speaker, the broadcast from the internal speaker network and the television on the wall. Two people often talk at the same moment.
Recognizers are built to transcribe one voice at a time. Given overlapping speech, they tend to merge fragments of both into a single stream of words that nobody said, or to follow the louder voice and drop the quieter one, even when the quieter one gave the order. Separating speakers first, a step called diarization, helps when voices are distinct and the recording keeps each line on its own channel. It helps much less when everything has been mixed down to one track.
Where the jargon comes in
Accent and audio problems compound with vocabulary. A general language model that has never seen “cable”, “yard” or a clipped bond name has no prior reason to prefer them, so when the acoustic evidence is weak it falls back to common words. A trader with an accent the model knows well, saying ordinary words on a clean line, may be transcribed almost perfectly. The same trader quoting a price in desk shorthand over a speaker channel may not.
What firms can do
The first step is to measure on their own audio. A sample of real calls from each desk, transcribed by hand and scored against the machine, shows where the errors fall, and the breakdown by speaker often matters more than the average. A system that performs well on nine traders and badly on the tenth is not good enough for the tenth trader’s records.
Recording choices help too. Keeping each line and handset on a separate channel, storing audio at a higher sample rate where the equipment allows it, and avoiding heavy compression before transcription all give the recognizer more to work with.
Training data is the longer fix. Adapting the acoustic model with recordings from the firm’s own speakers, and the language model with its own vocabulary, narrows the gaps the PNAS study found. It does not close them entirely, which is why human review of any transcript that will be relied on, for a client complaint or a regulator’s request, remains standard practice.