November 17, 2026

Which speech-to-text models survive noisy real-world call audio?

Which speech-to-text models survive noisy real-world call audio?

A speech-to-text model that scores well on clean studio recordings can still fall apart the moment a caller dials in over a crackly phone line, a codec squeezes the audio, packets drop mid-sentence, or a colleague talks over them in the background.

Introduction

The models that hold up best under those conditions are the ones evaluated on degraded audio, not just clean clips, and the differences between them are measurable: the MISTT benchmark from Bluejay Labs tests 104 speech-to-text models on clean recordings plus degraded copies that introduce phone-line distortion, low-bitrate codecs, packet loss, background noise, and an intelligible background talker, reporting accuracy with confidence intervals so you can see which models genuinely separate from the pack.

Most speech-to-text leaderboards are built on clean audio. That is a problem if you are choosing a model for a voice agent, because real calls almost never sound like a recording booth. The practical question is not "which model has the lowest word error rate" in the abstract. It is "which model keeps that low error rate when the audio is impaired." This article explains what degrades speech recognition in real deployments, how to evaluate models under those conditions, and what the current evidence says about which models hold up.

Key Takeaways

  • Clean-audio accuracy is a poor predictor of production performance. Robustness has to be measured on paired clean and degraded audio that reproduces phone lines, codecs, packet loss, noise, and overlapping talkers.
  • Packet loss and background talkers are among the hardest impairments for speech recognition, because they remove or corrupt the exact acoustic evidence the model needs to identify words.
  • Accuracy numbers without uncertainty are misleading. Models whose confidence intervals overlap the leader's are statistically tied, not ranked below it.
  • Streaming and batch models should be evaluated separately, since streaming models face an additional latency constraint and are timed from the end of speech to the final transcript.
  • Once you have picked a model, keep testing it against impaired audio in your own agent's workflows, because benchmark robustness does not guarantee robustness on your callers' audio.

What Actually Breaks Speech Recognition in Production

Anything that distorts the acoustic input widens the gap between what the model hears and what it was trained on. The main culprits in voice deployments are:

Phone-line and codec distortion. Traditional telephony and low-bitrate codecs strip out frequency content, so words that were acoustically distinct on a wideband recording can become ambiguous after compression.

Background noise. Steady noise from fans, traffic, or music masks the acoustic detail that distinguishes similar-sounding words.

Packet loss. When network packets drop, chunks of audio vanish or arrive corrupted. The model has to infer missing words from context, , which quietly degrades downstream understanding even when the transcript still looks roughly right.

A second person talking. An intelligible background talker is arguably the hardest case, because the interference is itself speech. The model cannot simply filter out "noise"; it has to decide which voice is the speaker, and overlapping speech actively competes with the target words.

Each impairment hits different models differently: some degrade gracefully, while others drop sharply. That variance is why testing on degraded audio matters more than testing on clean audio.

How to Evaluate Models Under Impaired Audio

The method that produces trustworthy answers is straightforward: take the same recordings, create clean and degraded versions of each, and measure how much accuracy each model loses between the two. That is the design behind the MISTT benchmark from Bluejay Labs, which runs 104 models (57 batch, 47 streaming) across 2,999 clean clips and 400 degraded copies drawn from that pool, covering phone lines, low-bitrate codecs, packet loss, background noise, and an intelligible background talker. Because the clean and degraded versions are paired, the benchmark isolates the effect of the channel rather than confounding it with differences in content.

Two details of the methodology are worth copying into any evaluation you run yourself:

Report accuracy with confidence intervals. A model at 3.4% word error rate and a model at 3.5% are usually not different in any meaningful sense. The MISTT leaderboard treats models whose intervals overlap the leader's as statistically tied rather than ranked, which prevents you from buying a distinction that does not exist. For reference, the published lowest English word error rates on the MISTT speech-to-text leaderboard are 3.33% for the batch model MAI-Transcribe-2 and 3.43% for the streaming model MAI-Transcribe-2 Streaming, with several models tied with each leader.

Time streaming models from end of speech. Streaming models face a latency budget that batch models do not, and a model that transcribes accurately but delivers the final transcript seconds late will break a real-time voice agent. Measuring latency from the forced end of speech to the final transcript reflects that constraint.

Which Models Hold Up Best

The honest answer from the current evidence: the top of the clean board is a statistical tie, and degradation reshuffles it: only five of the clean top ten stay in the top ten under severe impairment, and three under crosstalk. The gap that matters is between that top group and the long tail of models that look fine on clean audio but lose substantially more accuracy on impaired audio. That long tail is where real-world failures live, so compare per-model results across impairment categories rather than a single blended number: a model that is robust to noise but fragile to packet loss is a poor fit for a flaky network, and vice versa.

From Benchmark to Your Callers

A benchmark tells you which models degrade gracefully in general. It cannot tell you how a model handles your callers' specific audio: your telephony stack, your codecs, your acoustic environments, your languages and accents. Robustness is a property of the whole pipeline, and it drifts as you change providers, carriers, or models.

That is where ongoing testing earns its keep. Bluejay is an AI quality platform for testing and monitoring conversational agents, and it measures the audio conditions that break speech recognition directly: among its 27 speech-quality metrics are noise, packet loss, clipping, dropouts, and clarity, captured on both the caller and agent channels, with latency reported at P50/P95/P99 broken down by STT, LLM, and TTS. That means you can see whether the speech-to-text layer in your own stack is degrading on real calls, not just in a lab. To go deeper on how these checks fit into a testing workflow, see the Bluejay docs.

The practical workflow: pick a model whose degraded-audio performance is strong and statistically separated from the field, then verify it continuously against your own production audio, so that a codec change or a carrier update never silently regresses your transcription quality.

Frequently Asked Questions

Does background noise really change which speech-to-text model is best?

Yes. Models differ substantially in how much accuracy they lose on degraded audio, and those differences are invisible on clean-audio leaderboards. A model that leads on clean recordings can fall behind several alternatives once noise, packet loss, or an overlapping talker is introduced, which is why robustness testing on paired audio is the reliable way to compare them.

Which impairment is hardest for speech recognition: noise, packet loss, or another talker?

An intelligible background talker is generally the hardest, because the interference is itself speech and competes directly with the target words. Packet loss is also severe because it removes audio entirely, forcing the model to guess missing words. Steady background noise is disruptive but often more recoverable, since it masks rather than replaces speech.

Why do benchmarks report confidence intervals instead of a single ranking?

Because small word-error-rate differences are frequently within measurement uncertainty. When two models' confidence intervals overlap, the data does not support saying one is better than the other. Treating overlapping models as statistically tied, as the MISTT methodology does, prevents decisions based on differences that would not reproduce on your own audio.

How do I keep transcription quality from regressing after deployment?

Monitor the audio conditions that degrade recognition, including noise, packet loss, and dropouts, on live calls and alert when they cross thresholds. Bluejay tracks these speech-quality metrics on both channels of every conversation and reports latency by pipeline stage, so regressions in the speech-to-text layer surface in real time instead of as customer complaints.

Conclusion

The speech-to-text models that hold up best under background noise, packet loss, and overlapping talkers are the ones with proven robustness on degraded audio, not just strong clean-audio scores. Evaluate candidates on paired clean and impaired recordings, insist on confidence intervals, and separate streaming from batch models because they face different latency constraints. Then close the loop: Bluejay lets you test and monitor your voice agents under real-world audio conditions such as background noise, low-bitrate audio, and poor phone connections, so the model you chose keeps performing on the calls your customers actually make. Visit getbluejay.ai to see how it fits your stack.