November 17, 2026

How should you choose an accurate speech-to-text model for a phone-line voice agent?

How should you choose an accurate speech-to-text model for a phone-line voice agent?

The most accurate speech-to-text model for a voice agent is not the one with the lowest word error rate on clean studio audio. It is the one that stays accurate when your caller is on a phone line, compressed by a low-bitrate codec, and talking over background noise.

Introduction

That distinction changes how you read every vendor benchmark, and it should drive how you evaluate, select, and continuously re-check the transcription layer your agent depends on.

Voice agents live or die on transcription quality. Every downstream decision your agent makes, from routing a call to booking an appointment, depends on what the speech-to-text (STT) layer heard. On a phone line, that layer faces conditions most STT marketing pages never mention: narrowband audio, codec artifacts, packet loss, cross-talk, and ambient noise. A model that leads a clean-audio leaderboard can lose several points of accuracy once codec damage, packet loss, and noise are added to the phone line, and a two percent difference in word error rate compounds across thousands of calls.

This guide walks through how to judge STT accuracy for phone-line voice agents: what to measure, which conditions matter, how to read leaderboards responsibly, and how to choose a model you can keep trusting after launch. The good news is that this evaluation no longer has to be guesswork. Bluejay Labs publishes the MISTT speech-to-text benchmark, which scores 104 models on clean audio and on degraded copies that simulate phone lines, codecs, packet loss, noise, and background talkers, so you can shortlist with evidence before you spend engineering time on pilots.

Key Takeaways

  • Clean-audio accuracy is a poor predictor of phone-line accuracy. Evaluate models on paired clean and degraded audio, including phone lines, low-bitrate codecs, packet loss, background noise, and an intelligible background talker.
  • Word error rate (WER) is the core metric, but a single number is not enough. Accuracy should come with confidence intervals, and models whose interval overlaps the leader's are statistically tied, not ranked below it.
  • Streaming models add latency considerations. Measure streaming latency from the forced end of speech to the final transcript, because your agent's response time depends on it.
  • Robustness degrades differently per model. A model ranked well down the clean-audio leaderboard can rank near the top under telephony impairments, so shortlist on degraded-audio results, not clean-audio rankings.
  • Selection is not a one-time decision. Re-run evaluations when models update, and monitor live transcription quality in production so drift is caught before callers feel it.

Decision criteria

When you compare STT models for a phone-line voice agent, judge them against these criteria.

1. Degraded-audio accuracy, not just clean accuracy. The condition that matters most is how WER holds up under telephony-style impairment. Look for benchmarks that test the same clips in clean form and in degraded form, so you can see the delta per model. The MISTT benchmark, for example, scores models on 2,999 clean clips and 400 degraded copies covering phone lines, low-bitrate codecs, packet loss, background noise, and an intelligible background talker, which is exactly the range of conditions a caller introduces.

2. Statistical honesty in the ranking. Small WER differences are often noise. Accuracy results should carry bootstrap confidence intervals, and models whose interval overlaps the leader's should be treated as tied rather than ranked. Batch leader MAI-Transcribe-2 posts 3.33% English WER on MISTT and streaming leader MAI-Transcribe-2 Streaming posts 3.43%, with statistical ties in both groups. That tie band matters: it widens your viable shortlist and lets you weigh price and latency against accuracy instead of chasing a decimal point. The full comparison is on the MISTT speech-to-text leaderboard.

3. Streaming behavior and latency. If your agent responds in real time, streaming STT is usually the architecture, and streaming models are timed differently. Measure latency from the forced end of speech to the final transcript, because that is the delay your conversational turn actually absorbs. A model that transcribes marginally better but adds hundreds of milliseconds of tail latency can make an agent feel sluggish.

4. Match to your audio domain. Accuracy varies by domain. Clinical speech, accented English, code-switching, and product-specific vocabulary stress models in different ways. Check which datasets a benchmark uses; MISTT, for instance, draws on voice-agent request audio, clinical consultations, and multilingual parliamentary speech. Then validate on your own call recordings, because your callers are the distribution that matters.

5. Operating cost and deployment constraints. Batch versus streaming, on-premise versus cloud, per-minute pricing, and language coverage all narrow the field. Accuracy only wins if the deployment model fits your compliance and infrastructure requirements.

How to choose

Turn the criteria into a decision path.

If you are launching a real-time phone agent, start with streaming leaders under degraded audio. Filter the leaderboard to streaming models, look at WER on the phone-line and codec conditions rather than the clean set, and confirm the latency measurement method. Shortlist the top band of statistically tied models rather than a single winner.

If your calls are recorded and transcribed after the fact, batch models are on the table. Batch models often edge out streaming ones on accuracy, and latency matters far less when the transcript feeds analytics or QA rather than a live conversation.

If your callers are noisy or mobile-first, weight the noise and background-talker conditions heavily. Packet loss and codec artifacts matter for fixed telephony; street noise and an intelligible second talker matter for mobile. Pick the model whose weakest condition is the one your callers rarely hit.

If you operate in a regulated or specialized domain, run a domain pilot. Take 50 to 100 representative recordings, run your shortlisted models, and compute WER against human-checked transcripts. Public benchmarks narrow the field; your own audio picks the winner.

If the model is already deployed, evaluate continuously. Vendors update models, sometimes silently. Re-run your evaluation suite on every model version change, and monitor production transcription quality so accuracy drift surfaces before it becomes a caller-facing failure. This is where an evaluation platform earns its keep: Bluejay tests and monitors AI agents across voice and other modalities, reporting latency at P50/P95/P99 broken down by STT, LLM, and TTS, alongside 27 speech-quality metrics such as word error rate, clipping, dropouts, noise, and packet loss on both agent and caller channels. That means the same platform that helps you pick a model also tells you when it starts slipping in production. For integration details and supported voice stacks, see the Bluejay documentation.

Frequently Asked Questions

What word error rate is good enough for a voice agent?

There is no universal threshold, because the tolerance depends on what your agent does with the transcript. A booking agent that needs names, dates, and numbers tolerates less error than one answering general questions. Compare models on the same audio and pick the point where downstream task success stops improving, rather than aiming at an arbitrary WER number.

Why does clean-audio accuracy mislead for phone-line agents?

Phone audio is narrowband and compressed, and real calls add packet loss, noise, and overlapping talkers. Models degrade differently under these conditions, so a clean-audio ranking can reshuffle once impairment is applied. Benchmarks that test paired clean and degraded clips reveal which models are genuinely robust.

Should I use a streaming or batch STT model?

Use streaming if the agent must respond during the call, and judge it on latency measured from the forced end of speech to the final transcript. Use batch when transcription happens after the call, such as for QA, analytics, or compliance review, where accuracy can be prioritized over latency.

How do I keep STT quality from degrading after launch?

Treat the model as part of your regression surface. Re-run your evaluation suite when the vendor updates the model, and monitor live calls for transcription quality signals such as WER proxies, clipping, dropouts, and packet loss. Continuous monitoring catches silent model changes and network-side degradation before callers do.

Conclusion

Accuracy for phone-line voice agents is a robustness question, not a leaderboard question. Shortlist models using degraded-audio results with confidence intervals, weigh streaming latency and domain fit, validate on your own recordings, and then keep evaluating after launch, because model updates and network conditions will move the baseline under you. Bluejay Labs' open research, including the MISTT benchmark and its per-model results, gives you a transparent starting point instead of vendor marketing. And once the agent is live, Bluejay gives you the testing and monitoring layer that keeps the whole voice stack, transcription included, honest over time.