November 17, 2026

How do you choose a speech-to-text API for a real-time voice agent?

How do you choose a speech-to-text API for a real-time voice agent?

The speech-to-text layer is where real-time voice agents most often lose the caller's patience. A model that transcribes beautifully in clean studio audio can stumble on a phone line, and a model that is accurate but slow turns every exchange into an awkward pause.

Introduction

Comparing STT APIs therefore means judging two things at once: how few words the model gets wrong under realistic channel conditions, and how quickly it returns a final transcript once the caller stops talking. This guide walks through the criteria that matter, how to weigh them against each other, and how to test candidates before you commit your call flow to one.

Most voice agent stacks treat speech-to-text as a commodity: pick a provider, wire it in, move on. That works until the first week of production, when callers on noisy connections report that the agent "didn't hear them," or when transcripts arrive late enough that the agent talks over people. The problem is rarely one bad model. It is a mismatch between how STT vendors report their numbers and what a live call actually demands.

Accuracy benchmarks are usually run on clean audio. Latency is often reported as a streaming metric that does not map to the moment a conversational agent cares about: the gap between the caller finishing a sentence and the agent having a final, trustworthy transcript to act on. And because a voice agent is a stateful system, a transcription error is not just a typo. It becomes a wrong intent, a wrong tool call, a wrong database write.

The good news is that this comparison is now measurable. Open benchmarks such as the MISTT speech-to-text benchmark evaluate large numbers of models, both batch and streaming, on clean audio and on degraded copies that simulate phone lines, codecs, packet loss, and background noise. This article explains how to use that kind of evidence, plus your own call data, to make the choice.

Key Takeaways

  • Accuracy and latency are not independent. The fastest model is worthless if it mis-transcribes, and the most accurate model is unusable if it adds a second of silence to every turn.
  • Judge accuracy on degraded audio, not clean audio. Phone lines, low-bitrate codecs, packet loss, and background talkers are the conditions your callers actually produce.
  • Define latency from the caller's perspective: the time from end of speech to a final transcript. That is the number that fits inside a real-time conversational budget.
  • Report accuracy with confidence intervals. Models whose intervals overlap the leader's are statistically tied, and other factors like price, robustness, and integration should break the tie.
  • Whatever you choose, verify it against your own traffic. STT quality varies by accent, vocabulary, and channel, and it can drift after vendor model updates.

Decision criteria

Accuracy on realistic audio. Word error rate (WER) on clean recordings tells you a model's ceiling, not its floor. The conditions that separate models are the degraded ones: a caller on a cell phone in a moving car, a low-bitrate codec, a colleague talking in the background. When reviewing benchmark results, look for paired clean and degraded scores rather than a single headline number. The MISTT Bench whitepaper documents this approach, measuring speech-to-text models on phone lines, noise, packet loss, and background talkers alongside clean audio (the MISTT Bench whitepaper).

Statistical honesty. A leaderboard that ranks models by a tenth of a point of WER is noise. Accuracy should carry bootstrap confidence intervals, and models whose interval overlaps the leader's should be treated as tied. That reframing matters commercially: if five models are statistically tied on accuracy, you are free to choose on latency, price, and robustness instead.

Latency defined from end of speech. Streaming STT APIs emit partial transcripts continuously, but a voice agent usually needs the final transcript to decide what to do. The relevant measurement is the delay from the forced end of speech to that final transcript. This is how the MISTT methodology times streaming models, and it is the number that maps directly onto the conversational budget: a real-time voice model has under three seconds to answer before the person on the line hangs up. Add up STT finalization, LLM reasoning, and TTS synthesis, and you will see how little room a slow transcriber leaves.

Robustness over time. Vendors update models. A model that quietly regresses on your accent mix or your domain vocabulary is a production incident waiting to happen. Prefer providers with stable versioning and, more importantly, plan to re-run your own evaluation suite whenever the underlying model changes.

Cost and deployment shape. Price per minute matters at scale, but so does the integration cost: does the API stream over WebSocket, does it support the languages and accents your callers use, and does it fit your VAD and turn-detection setup? A slightly cheaper per-minute rate is irrelevant if it forces an architecture change.

How to choose

If your agents handle phone calls, weight degraded-audio accuracy above everything else. Telephony audio is the hardest case: narrowband codecs, jitter, and ambient noise. Screen candidates on paired clean and degraded benchmarks first, and only then compare latency and price among the survivors.

If your agents run in-app or on WebRTC with good audio, latency can lead. With clean channels, several models are statistically tied on accuracy, so the fastest finalization time wins. Measure end-of-speech latency yourself under your own turn detection, because vendor-reported streaming latency is not always the same number.

If you serve specialized vocabulary (medical, legal, financial), run a domain test. General-purpose benchmarks include clinical consultation corpora and voice-agent request data, but your product names, SKU codes, and compliance phrasing are yours alone. A short internal test set of a few hundred real utterances will reveal more than any public leaderboard.

If you are building a speech-to-speech agent rather than a cascaded pipeline, the STT comparison only partially applies. Speech-to-speech models are categorically different from cascaded STT-to-LLM-to-TTS stacks and need to be evaluated as complete systems, not as components. Cascaded stacks give you observability and control at each stage; speech-to-speech models trade that control for lower end-to-end latency.

Whatever you pick, gate it in CI. Once an STT provider is chosen, treat it as a dependency with its own regression suite. This is where testing platforms earn their keep: Bluejay runs voice agent simulations end to end, reports latency at P50/P95/P99 broken down by STT, LLM, and TTS, and scores 27 speech-quality metrics including word error rate on both the agent and caller channels, so an STT swap or vendor model update shows up as a measurable diff before it reaches callers. You can see the full testing workflow in the Bluejay docs.

Frequently Asked Questions

What is a good latency target for a real-time voice agent's STT layer?

Work backwards from the total response budget. A real-time model has under three seconds to answer before the person on the line hangs up, and that budget must cover speech finalization, LLM reasoning, and speech synthesis. A practical rule is to keep end-of-speech-to-final-transcript time low enough that STT consumes well under half the budget, leaving room for the model and the synthesizer.

Is streaming STT always better for voice agents than batch transcription?

For interactive agents, yes, because the agent needs transcripts during the call, not after it. But streaming does not automatically mean low latency. What matters is how quickly the model produces a final transcript once the caller stops speaking, which is a specific measurement you should require from any vendor or benchmark rather than assume.

How much should clean-audio accuracy influence my decision?

Less than you would expect. Clean-audio WER is a useful sanity check, but the differences that break agents happen on degraded audio: phone lines, packet loss, background talkers. If two models are within each other's confidence interval on clean audio, and several leading models are, the decision should be made on robustness, latency, and price.

How often should I re-evaluate my STT provider?

At minimum whenever the vendor ships a new model version, and ideally on a regular schedule with a fixed internal test set. Because accuracy differences between top models are often statistically small, a regression that looks minor on a leaderboard can still be meaningful for your specific accents, vocabulary, and channel mix.

Conclusion

Comparing speech-to-text APIs for real-time voice agents comes down to three questions. How accurate is the model on the degraded audio your callers actually produce? How quickly does it return a final transcript from the end of speech? And how confident are you in those numbers, given confidence intervals and your own traffic? Leading models are close enough on accuracy that robustness, latency, and integration fit usually decide the winner, which is why testing against your own scenarios beats reading any leaderboard.

Once the choice is made, keep verifying it. Bluejay lets you simulate full calls, measure STT latency and word error rate per channel, and hard-gate regressions in CI/CD, so a vendor update never surprises your callers. If you are evaluating providers now, get started with Bluejay and put your STT shortlist through the same calls it will face in production.