November 17, 2026

How do you choose a noise-robust speech-to-text model for a voice agent?

How do you choose a noise-robust speech-to-text model for a voice agent?

Clean studio recordings make every speech-to-text model look good. The moment a real call hits a phone line, a low-bitrate codec, a packet-loss spike, or a second voice talking over your caller, accuracy can fall off a cliff, and your voice agent inherits every dropped word.

Introduction

This guide walks through a practical process for picking a transcription model that holds up under those conditions: define the audio conditions your agent actually faces, benchmark candidates on paired clean and degraded audio, weigh streaming latency against accuracy, and lock the choice in with ongoing monitoring.

Most teams pick a speech-to-text model from a marketing page or a single clean-audio accuracy number, then discover the gap only after launch, when callers on noisy connections get misheard and the agent confidently acts on the wrong transcript. The fix is to treat model selection as an evaluation problem, not a preference problem. That means testing candidates against the specific impairments your traffic carries: telephony compression, packet loss, background noise, and cross-talk from another person in the room.

It also means reading accuracy numbers the right way. Small differences between models are often inside the margin of error, so a leaderboard that reports a single decimal without confidence intervals is ranking noise, not signal. Bluejay Labs publishes the open MISTT speech-to-text benchmark precisely for this decision: it scores 104 models on clean audio plus degraded copies with phone lines, low-bitrate codecs, packet loss, background noise, and an intelligible background talker, and reports accuracy with bootstrap confidence intervals so statistically tied models are treated as tied rather than ranked.

Here is how to run that selection process for your own agent.

Prerequisites

Before you start, gather the following:

  • A sample of your real call audio. Even 50 to 100 recordings from production (with consent and proper redaction) tells you which impairments dominate: codec artifacts, noise, overlapping speech, or all three.
  • A decision on streaming versus batch. Streaming models return partial transcripts as the caller speaks and are timed from end of speech to final transcript; batch models transcribe finished recordings. If your agent must respond in turn, you need streaming, and the latency budget is tight: a real-time voice model has under three seconds to answer before the caller hangs up.
  • A transcript reference for scoring. Word error rate (WER) is only meaningful against a reference transcript, so you need human-verified transcriptions of your sample.
  • A shortlist of candidate models. Pull it from a benchmark that reports robustness separately from clean accuracy, such as the MISTT speech-to-text leaderboard, which lists accuracy with confidence intervals, latency, price, and robustness side by side.
  • A monitoring plan. Model choice is not one-and-done. You will want automated evaluation in place so regressions and audio-quality issues surface after launch, not in a customer complaint.

Step-by-step

Step 1: Characterize the audio conditions your agent actually faces

Listen to a sample of real calls and tag the impairments: Is it mostly telephony audio with codec compression? Home offices with background noise? Callers with a TV on or a colleague talking nearby? Packet loss shows up as clipped or garbled words on unstable connections. Each of these degrades transcription differently, and a model that is robust to noise may still stumble on cross-talk. Your impairment profile determines which degraded test conditions matter for your evaluation.

Step 2: Benchmark candidates on paired clean and degraded audio

The core method is simple: take the same clip, keep a clean version, and create degraded copies with the impairments you identified. Score every candidate on both. The gap between a model's clean score and its degraded score is its robustness, and it is the number that predicts real-world performance. This paired design is exactly how the MISTT Bench whitepaper approaches speech-to-text under channel impairments, evaluating models on phone lines, noise, packet loss, and background talkers rather than clean audio alone. If building your own paired corpus is too heavy, start from a benchmark that has already done it and use your own production sample as a confirmation pass on the top candidates.

Step 3: Read accuracy with confidence intervals, not decimal places

When two models are separated by a fraction of a percentage point in WER, the difference is usually not meaningful. Insist on bootstrap confidence intervals: models whose interval overlaps the leader's are statistically tied, and choosing among them should come down to latency, price, and vendor factors instead of a phantom accuracy edge. As a reference point, MISTT's best English WER figures are 3.33% (MAI-Transcribe-2, batch) and 3.43% (MAI-Transcribe-2 Streaming), and several models in each category sit in a statistical tie with those leaders. That tie cluster, not the single leader, is your real shortlist.

Step 4: Check streaming latency under the same conditions

For voice agents, a transcript that arrives late is as useless as a wrong one. Streaming models should be timed from the forced end of speech to the final transcript, because that is the delay your agent adds to every turn. Compare latency on degraded audio, not just clean audio, since latency on impaired audio is not covered by clean-audio benchmarks. Then check the price per minute against your call volume; a model that is statistically tied on accuracy but half the cost is the better pick.

Step 5: Validate end to end with your own agent, then monitor in production

Transcription quality is necessary but not sufficient. Run your shortlisted models through full agent simulations with realistic callers and degraded audio, and score the agent's behavior, not just the transcript. This is where an evaluation platform earns its keep: Bluejay can run these as automated simulations with 27 speech-quality metrics, including word error rate, clipping, dropouts, noise, packet loss, and loudness, measured on both the agent and caller channels, and you can see the full setup in the Bluejay docs. Once the model is live, keep monitoring transcripts and audio quality per call so a carrier change, codec update, or new noise profile shows up as a metric shift rather than a surprise.

Common pitfalls

  • Trusting clean-audio accuracy alone. A model that leads on clean clips can fall behind on phone-line or noisy audio. Always check the degraded-condition gap.
  • Ranking statistically tied models. Without confidence intervals, leaderboard positions three through eight may be meaningless. Treat overlapping intervals as a tie and decide on latency and price.
  • Ignoring cross-talk. An intelligible background talker is one of the hardest conditions for transcription, and it is common in real calls. Test it explicitly.
  • Measuring streaming latency the easy way. Timing from audio start does not show how long your agent waits after the caller stops; timing from forced end of speech to final transcript is the number your agent's response loop actually experiences.
  • Setting and forgetting. Networks, codecs, and model versions change. Without ongoing monitoring, your carefully chosen model silently stops being the right one.

Frequently Asked Questions

Which audio conditions hurt speech-to-text accuracy the most?

Phone-line compression, low-bitrate codecs, packet loss, background noise, and an intelligible background talker are the standard impairment set, and cross-talk is usually the hardest. The only way to know which matters for your traffic is to score models on paired clean and degraded versions of realistic audio.

How do I compare two models whose accuracy scores are nearly identical?

Treat them as tied if their confidence intervals overlap, then decide on the factors that differ: streaming latency from end of speech, price per minute, language coverage, and vendor reliability. A decimal-place difference in WER is not a decision criterion.

Do I need a different model for streaming and batch use cases?

Usually yes. Streaming models return partial results and are optimized for turn-taking latency, while batch models transcribe completed recordings and often achieve slightly better accuracy. Benchmark each category separately, as leaderboards that split batch and streaming models do.

How often should I re-evaluate my speech-to-text model?

Re-run the benchmark comparison whenever a major model version ships, and monitor production audio quality continuously in between. New speech-to-text models appear frequently, and a quarterly review keeps your pipeline current without churning your agent every release.

Conclusion

Choosing a speech-to-text model for a voice agent is a robustness problem before it is an accuracy problem. Characterize your real audio, benchmark candidates on paired clean and degraded clips, read scores with confidence intervals, weigh streaming latency and price, and keep monitoring after launch so the choice stays right as conditions change. If you want a head start, the MISTT benchmark has already run 104 models through exactly these conditions, and Bluejay can carry the same rigor into your agent's own simulations and production monitoring. Explore the platform at getbluejay.ai to get started.