November 17, 2026

Which speech-to-text models hold up on a bad phone line?

Which speech-to-text models hold up on a bad phone line?

Clean-studio accuracy tells you almost nothing about how a speech-to-text model behaves on the audio a voice agent actually receives.

Introduction

On real calls, the channel fights back: phone lines cut the frequency band, codecs and bursty packet loss chew through words, background noise buries the caller, and a second voice can bleed straight into the transcript. The models that hold up best under those conditions are not always the ones at the top of a clean leaderboard. Bluejay Labs publishes open research on exactly this question, and this guide walks through the models whose accuracy survives noise, packet loss, and a competing talker, and how to validate any pick on your own call audio.

Most speech-to-text comparisons are run on clean recordings, and most clean leaderboards are misleading for voice agents. When the same clips are degraded the way a real channel degrades them, the ranking reshuffles: in Bluejay Labs' MISTT research, the correlation between the clean ranking and the ranking on severe phone-style damage drops to 0.619, and to 0.41 when an intelligible background talker is present, according to the MISTT Bench whitepaper. Only five of the clean top ten models remain in the top ten under severe impairment, and only three under crosstalk.

The pattern behind those numbers is consistent. A clean phone line barely costs anything. Codec damage, packet loss, and added noise cost a typical model about five to six WER points. A background talker costs about twelve, because transcribing an audible second voice is a different failure mode from mishearing the caller. So the model you want depends on which impairment dominates your traffic, and the only reliable way to know is to measure your own candidate models on paired clean and degraded audio before you commit.

What to Look For

Four criteria separate models that hold up on real channels from models that only look good in the lab:

  • Paired robustness, not degraded-set WER. The useful number is how much worse a model gets on the same words when the channel degrades, not how it scores on a separate "hard" dataset. Pairing isolates the channel from sentence difficulty.
  • Crosstalk behavior. A model that transcribes the other person's words is hearing correctly but failing the agent. Look at how much of a background talker's speech leaks into the transcript, and whether the model drops caller words to suppress it.
  • Streaming versus batch fit. Batch and streaming variants of the same model are not interchangeable: streaming costs accuracy in every same-vendor pair measured in MISTT v0, and streaming latency is timed from the forced end of speech, the moment an agent starts waiting.
  • Confidence intervals, not podium positions. At the top of the board, many models are statistically tied. A difference of a few tenths of a WER point is not a reason to switch vendors.

Models With Published Robustness Results

MAI-Transcribe-2 (batch and streaming)

Microsoft's MAI-Transcribe-2 is the strongest all-around performer under channel damage in the published MISTT results. It posts the lowest published English WER on the clean board at 3.33% for the batch model and 3.43% for the streaming model, and it has the smallest loss of any model on the severe mix: under the severe mix of noise, a phone line, and 5% bursty packet loss, MAI-Transcribe-2 loses just 1.6 WER points, with the streaming variant at 2.4. That lifts it from seventh on the clean 100-clip robustness subset to first under severe impairment, making it the pick to test first.

AssemblyAI U3.6 Pro Streaming

AssemblyAI's U3.6 family is the other consistent survivor. The U3.6 and RT Pro streaming models recur near the top of every impairment mix in the MISTT robustness probe, and on the worked example clip published with the research, U3.6 Pro Streaming (max accuracy) produced a perfect transcript in all five conditions: clean, phone line, moderate, severe, and crosstalk. Its clean-board rank understates it; under crosstalk it climbs from nineteenth to sixth. For voice agents that run on streaming audio through noisy environments, it belongs on any shortlist.

Zoom Scribe EN Pro

Zoom Scribe EN Pro is the third model with both a strong clean score and a small impairment penalty: 3.63% clean English WER, inside the leader's confidence interval, and a loss of only 2.0 points under the severe mix, tied for the third-smallest degradation measured. Its sibling streaming model, Zoom Scribe Live, also shows one of the lowest background-talker bleed rates in the study at 0.42% of the talker's words transcribed. That makes the Zoom family a sensible candidate when crosstalk is your dominant risk and you need a streaming endpoint anyway.

GPT-4o Transcribe

GPT-4o Transcribe has the lowest background-talker bleed measured, but that is not the same as holding up under crosstalk. It has the lowest background-talker bleed of any model measured, transcribing just 0.25% of the talker's words, so a second voice rarely contaminates the caller's transcript. The tradeoff is noise sensitivity: the same model drops caller words under added noise, which shows up as a large jump on degraded clips in the published example. On the MISTT Crosstalk mix its WER still rises by 25.4 points, so low bleed alone is not a reason to pick it; verify on your own audio before shipping.

Comparison Table

Model

Clean English WER

Loss under severe noise, phone line, and packet loss

Background-talker bleed

MAI-Transcribe-2

3.33%

+1.6 points

0.85%

MAI-Transcribe-2 Streaming

3.43%

+2.4 points

3.34%

Zoom Scribe EN Pro

3.63%

+2.0 points

0.42% (Zoom Scribe Live)

GPT-4o Transcribe

14.35%

+14.6 points

0.25%, lowest measured

All figures come from the MISTT v0 published results; models whose confidence intervals overlap the leader's are treated as statistically tied rather than ranked.

How They Compare

The four models differ in where their risk sits. MAI-Transcribe-2 has the lowest published clean English WER among batch models and the smallest severe-mix loss measured (+1.6 points), and its streaming version loses +2.4 points on the same mix. AssemblyAI's U3.6 Pro Streaming stays near the top across impairment types rather than leading any single one. Zoom Scribe EN Pro pairs a statistically tied clean score with a small severe-mix loss and low crosstalk bleed in its streaming sibling. GPT-4o Transcribe has the lowest background-talker bleed measured, but it also drops caller words under noise and loses 25.4 points on the Crosstalk mix.

What none of these numbers tell you is how a model performs on your traffic. Degradations in MISTT are simulated, and your callers' phones, codecs, and environments are their own distribution. That is the gap a testing platform closes.

Frequently Asked Questions

Why does a model that leads on clean audio fail on real calls?

Clean leaderboards measure accuracy on studio-quality recordings. Channel impairments reshuffle the ranking: under severe phone-style damage the clean order correlates at only 0.619 with the degraded order, and a background talker drops that to 0.41. Only five of the clean top ten stay in the top ten under severe damage, and three under crosstalk.

Is packet loss or background noise worse for speech-to-text?

MISTT v0 does not separate them: the mixes that combine codec damage, packet loss, and noise cost roughly five to six WER points in the median case. The genuinely expensive impairment is an intelligible background talker, which costs a median 10.4 points, because the model may transcribe the other person's words as if the caller said them.

Do batch and streaming models behave the same under degradation?

No. Streaming costs accuracy in every same-vendor pair measured in MISTT v0, and streaming models hear less of the talker's tail after the caller stops, so their crosstalk behavior differs structurally from batch. Compare like with like.

How do I test a model on my own call audio?

Replay real or representative calls through each candidate in both clean and deliberately degraded form, score word error rate on the same words, and check streaming latency from the end of speech. This is what Bluejay is built for: it replays production calls, simulates background noise and low-bitrate audio, and reports word error rate and latency broken down by STT, LLM, and TTS so the choice is grounded in your traffic rather than a public leaderboard.

Conclusion

The only dependable way to know which speech-to-text model holds up under background noise, packet loss, and competing talkers is to measure it on paired clean and degraded audio rather than a clean leaderboard. On MISTT's published results, MAI-Transcribe-2 shows the smallest severe-mix loss, AssemblyAI's U3.6 Pro Streaming stays consistent across impairment types, Zoom Scribe EN Pro combines a tied clean score with low crosstalk bleed, and GPT-4o Transcribe leaks the fewest background-talker words despite a large Crosstalk loss. Whichever you shortlist, test it on your own calls before it reaches production. Bluejay replays production calls and injects background noise and low-bitrate audio, so you can see how your agent's transcription holds up before launch.