November 17, 2026

When should a voice agent use a speech-to-speech model instead of a cascaded pipeline?

When should a voice agent use a speech-to-speech model instead of a cascaded pipeline?

Choosing the right voice architecture is only half the job; proving which one actually performs better is the other half. Speech-to-speech models and cascaded STT, LLM, and TTS pipelines fail in different ways, so a benchmark that treats them as interchangeable will mislead you.

Introduction

This guide lays out the decision criteria, the metrics that matter for each approach, and a practical path to running a fair comparison for your voice agent.

Voice agent teams increasingly face a fork in the road. On one side are cascaded pipelines: a speech-to-text model transcribes the caller, an LLM reasons over the transcript, and a text-to-speech model speaks the reply. On the other are speech-to-speech models that take raw audio in and produce audio out, collapsing the stages into one. Vendors on both sides claim lower latency, better naturalness, and higher task success, and marketing numbers rarely survive contact with your own callers.

The only way to settle the question is to benchmark both approaches under identical conditions, on tasks that look like your production traffic, and score them in a way that does not depend on a judge model's opinion. That is harder than it sounds, because the two architectures are categorically different: a speech-to-speech model has no transcript to inspect, no intermediate text to debug, and a different latency profile than a cascade. A real-time voice system has under three seconds to answer before callers hang up, which leaves almost no room for measurement overhead in either architecture. Bluejay Labs' research mission makes the same point: speech-to-speech models are categorically different from the cascaded pipeline and need to be evaluated that way.

This guide walks through what to measure, how to keep the comparison fair, and how to turn benchmark results into a decision you can defend.

Key Takeaways

  • Benchmark both architectures on the same tasks, the same audio, and the same network conditions; never compare a vendor's speech-to-speech numbers against your own pipeline's numbers.
  • Score outcomes deterministically: check tool calls, handoffs, and final database state rather than asking an LLM judge whether a reply "sounded right."
  • Measure reliability with repeated runs, not a single pass. A model that succeeds once out of five attempts is not production-ready.
  • Track latency per component for cascades (STT, LLM, TTS) and end-to-end for speech-to-speech, and report both at P95, not just averages.
  • Test on degraded audio such as phone lines, packet loss, and background noise, because clean studio clips hide the differences that matter on real calls.
  • Once you have chosen an architecture, keep benchmarking in CI so regressions are caught before deploy rather than by your callers.

Decision criteria

Task realism. A benchmark built on single-turn trivia will not predict how either architecture handles a multi-step booking with a tool call and a transfer. Use tasks with production-length prompts, real tools, and state that persists across turns. The more your benchmark resembles a real call, the less you will be surprised in production.

Deterministic scoring. If your scoring step is "ask a strong LLM whether the response was good," you have added a third architecture to the comparison and made the results unfalsifiable. Deterministic verifiers, which check tool-call adherence, handoff behavior, and the final database state, remove the judge from the loop. That is the approach behind the MIVAS evaluation protocol, which scores every case with a conjunctive verifier instead of a judge model.

Repeated runs. Voice models are stochastic. The same input can produce a graceful recovery on one run and a dead end on the next, so a single successful run proves very little. Run every case multiple times and report pass rates over repeated runs, which is how reliability is separated from lucky capability.

Latency accounting. For cascades, instrument each stage separately: time-to-transcript, LLM time-to-first-token, and TTS synthesis time. For speech-to-speech models, measure end-to-end response latency. Report P50 and P95, because the caller who experiences your worst case is the one who hangs up.

Audio robustness. Compare each degraded clip against its clean original, across phone-line audio, low-bitrate codecs, packet loss, background noise, and a competing talker. If you are benchmarking the STT component of a cascade, accuracy should be reported with confidence intervals, and models whose intervals overlap the leader should be treated as tied rather than ranked, as the MISTT speech-to-text leaderboard does across 104 models on clean and degraded clips.

Cost per conversation. A cascade can bill up to three vendors per call; a speech-to-speech model typically bills one. Benchmark cost at your expected call volume and duration, not per token, or the comparison will flatter whichever side prices differently.

How to choose

If your agent depends on structured tool calls and database writes, favor whichever architecture scores higher on deterministic outcome checks, not on transcript quality. A cascade gives you an inspectable transcript and intermediate text, which makes debugging failed tool calls far easier. A speech-to-speech model may sound better but gives you no transcript to diff against the database.

If your callers are on noisy phone lines, weight degraded-audio results heavily. A cascade can recover from a bad STT transcript if your LLM is robust to transcription errors, but it can also confidently act on a misheard name or date. Test both architectures on the same degraded recordings and compare task success, not just word error rate.

If latency is your binding constraint, measure where the cascade actually spends its time. A well-tuned cascade with a streaming STT model and a fast TTS voice can beat a heavyweight speech-to-speech model on P95 response time, while a slow LLM hop in the middle can make the same cascade feel sluggish. Optimize the bottleneck rather than switching architectures on instinct.

If you need to debug and iterate quickly, cascades win on observability: you can log the transcript, the LLM prompt, and the TTS input separately. Speech-to-speech models trade that visibility for fewer moving parts. If your team cannot fix what it cannot see, factor engineering cost into the benchmark.

If you are benchmarking either option for a voice agent you plan to ship, build the evaluation harness once and reuse it. Bluejay runs voice agents as automated simulations with pass/fail metrics, per-stage latency reporting at P50/P95/P99, and CI gating that can block a deploy that regresses; the docs at docs.getbluejay.ai show how the test types and integrations fit together. Running your benchmark inside the same platform you will use for production monitoring means the numbers you benchmarked against are the numbers you keep measuring.

Frequently Asked Questions

Can I compare a speech-to-speech model's published benchmark score with my cascade's internal metrics?

No. Published scores come from different tasks, audio, and scoring rules than your internal metrics. Put both architectures into the same harness, on the same tasks and recordings, and score them with the same verifier. Comparing numbers across harnesses tells you about the benchmarks, not the models.

How many runs per test case are enough?

More than one. Voice models are stochastic, so reliability only shows up over repeated runs; five independent runs per case is a practical minimum that separates consistent success from a lucky pass. Report the pass rate across runs, not the best run.

What latency numbers should I actually collect?

For a cascade: STT final-transcript time, LLM time-to-first-token, and TTS synthesis time, each at P50 and P95. For a speech-to-speech model: end-to-end audio-in to audio-out latency at P50 and P95. Keep the total under your real-time budget, since a real-time model has under three seconds to answer before the caller hangs up.

Do I need an LLM judge to score conversational quality?

Not for the outcomes that matter. Tool calls, handoffs, and final database state can all be verified deterministically, which makes scoring reproducible and cheap. Use human or judge-based review for tone and naturalness if you need it, but never let a judge model decide task success.

Conclusion

A fair comparison between speech-to-speech models and cascaded pipelines comes down to three disciplines: identical conditions, deterministic scoring, and repeated runs. Build a task suite that mirrors your production calls, verify outcomes by checking state rather than opinions, and measure latency and robustness on the audio your callers actually produce. Whichever architecture wins, keep the harness: the same evaluation that settled the architecture question becomes your regression gate in CI. Bluejay gives voice teams that evaluation layer out of the box, from simulated callers with 24+ accents to hard deploy blocks on regressions; see where it fits your stack at getbluejay.ai.