November 17, 2026

How should you benchmark speech-to-speech models against cascaded voice pipelines?

How should you benchmark speech-to-speech models against cascaded voice pipelines?

Benchmarking a speech-to-speech model against a cascaded pipeline of speech-to-text, an LLM, and text-to-speech means running both architectures through the same realistic, task-based evaluations and comparing them on task success, reliability, latency, and robustness, rather than on component-level quality alone.

Introduction

The two approaches are not interchangeable: a speech-to-speech model hears raw audio and speaks back in one model, while a cascaded pipeline hands off between three specialized stages, and each architecture fails in different ways. A fair benchmark isolates those differences with identical tasks, identical callers, and identical scoring, so the comparison reflects the architecture rather than the test setup.

Voice agent teams increasingly face a build-or-buy decision at the model layer: adopt an end-to-end speech-to-speech model, or assemble a cascaded pipeline from a speech-to-text model, a language model, and a text-to-speech voice. Vendor claims on both sides are hard to compare because they measure different things. The only defensible path is to benchmark both architectures under identical, production-shaped conditions.

This article explains how to design that comparison: what to measure, how to keep the playing field level, and where the architectures genuinely differ. It draws on evaluation practices published by Bluejay Labs' open voice AI benchmarks, which cover both speech-to-speech agents and the speech-to-text layer that cascaded pipelines depend on.

Key Takeaways

  • Compare architectures on end-to-end task outcomes, not component metrics; a great STT word error rate says nothing about whether the pipeline completes the caller's goal.
  • Measure reliability with repeated runs, not single passes, because voice agents are stochastic and a one-shot score hides flakiness.
  • Hold latency to a real-time budget and report it at the tail (P95/P99), where dropped calls actually happen.
  • Test robustness on degraded audio, since phone lines, noise, and packet loss affect the two architectures differently.
  • Use deterministic scoring of tool calls, handoffs, and final state so results do not depend on a judge model's opinions.

What makes speech-to-speech and cascaded pipelines different systems

A cascaded pipeline is modular. The speech-to-text stage transcribes the caller, the LLM reasons over text, and the text-to-speech stage speaks the reply. Each stage can be swapped, tuned, and measured independently, and the transcript gives you an audit trail for free. The cost is compounding latency and error propagation: a transcription mistake becomes a reasoning mistake, then a spoken mistake.

A speech-to-speech model processes audio in and audio out within a single model. It preserves prosody, interruption, and tone, and it removes two handoff points from the latency chain. The tradeoff is opacity: there is no intermediate transcript to inspect, so evaluation has to judge the conversation and its outcomes directly rather than by reading the pipeline's internals.

Because the architectures differ structurally, benchmarking them means evaluating the whole system against the caller's goal, not benchmarking each part in isolation.

Design the benchmark around end-to-end tasks

The unit of comparison should be a realistic task: book an appointment, resolve a billing dispute, complete an intake call. Each task places the agent inside a scenario with production-length prompts, tools, and isolated state, and a simulated caller with a goal. This is the approach behind the MIVAS evaluation protocol, which runs stateful voice-agent tasks across healthcare, legal, and customer support industries with both realtime speech-to-speech models and cascaded pipelines under identical conditions.

Wrap each architecture in a harness that exposes the same interface: the same tools, the same system prompt, the same caller simulation. If the cascaded pipeline gets a better STT model than the speech-to-speech model's built-in audio handling, that is a legitimate architectural difference, but note it explicitly so readers of the benchmark know what they are comparing.

Score with deterministic verifiers, not a judge model

An LLM-as-judge scorer introduces its own variance and can favor verbose or plausible answers. Deterministic verification is stricter and reproducible: check whether the right tool was called with the right arguments, whether the handoff happened at the right point, and whether the final database state matches the expected outcome. Combining these checks conjunctively (all must pass) gives a conservative pass/fail per run.

This is how MIVAS's deterministic conjunctive verifier scores every case: database-state diffing, tool-call adherence, and handoff adherence, combined with AND. Deterministic scoring matters especially when comparing architectures, because a judge model can mask the error-propagation differences you are trying to measure.

Measure reliability with repeated runs

Voice agents are stochastic. The same task can pass on one run and fail on the next because of a misheard word, a tool timeout, or a different conversational path. A single pass@1 number measures capability on a good day; reliability requires running every case multiple independent times and reporting how often it passes.

Run each task at least five times per architecture and report both single-run success and repeated-run success. The gap between the two numbers is often where the architectures diverge most: a pipeline with a brittle STT stage may match a speech-to-speech model on best-case runs while failing far more often across repeated ones.

Hold both architectures to a real-time latency budget

A caller will not wait long. Voice systems have a latency budget of roughly under three seconds before the interaction feels broken, and it applies end to end, not per component. For cascaded pipelines, measure the sum of STT finalization, LLM response, and TTS synthesis, and report P50, P95, and P99 rather than averages. For speech-to-speech models, measure time from end of caller speech to audible response start.

Tail latency is where architectures separate: a pipeline that averages 1.5 seconds may still have a P99 above budget when the STT stage waits for end-of-speech confirmation. Report the distribution, and flag any configuration that cannot hold the budget at the tail.

Test robustness on degraded audio

Clean studio audio flatters both architectures. Real calls arrive over phone lines with low-bitrate codecs, packet loss, background noise, and sometimes an intelligible background talker. Build a paired test set: the same clean clips plus degraded copies, so you can measure how much accuracy each architecture loses under each impairment type.

The two architectures degrade differently. A cascaded pipeline's failure is concentrated in its STT stage, which you can benchmark directly; the MISTT speech-to-text leaderboard measures 104 models on clean and degraded audio, reporting accuracy with bootstrap confidence intervals and streaming latency from end of speech. A speech-to-speech model's robustness has to be measured end to end, since there is no transcription stage to isolate. Either way, report accuracy with confidence intervals and treat models inside the leader's interval as statistically tied rather than ranked.

Put the results to work

Once the benchmark runs, use it as a regression gate rather than a one-time study. Wire the same tasks and verifiers into CI so every model upgrade, prompt change, or pipeline swap is re-scored before deployment. Bluejay is built for exactly this ongoing evaluation and monitoring of voice agents: it runs automated voice agent tests, reports latency at P50/P95/P99 broken down by STT, LLM, and TTS stage, and tracks 27 speech-quality metrics on both caller and agent channels, so the same harness that settled the architecture question keeps catching regressions after launch. To see how the platform fits an existing stack, visit getbluejay.ai or the developer documentation.

Frequently Asked Questions

Can I compare a speech-to-speech model and a cascaded pipeline on the same benchmark?

Yes, if both are wrapped in harnesses that expose the same tools, prompts, and caller simulation, and scored by the same deterministic verifiers. The comparison is then between architectures under identical conditions, not between two different test setups.

Why not just compare word error rates and TTS quality separately?

Component metrics hide error propagation and interaction effects. A pipeline can have excellent per-stage scores and still fail end-to-end tasks because of latency stacking or context loss between stages. Task-level outcomes are what callers actually experience.

How many runs do I need per task to measure reliability?

At least five independent runs per case is a practical floor for voice agents, since it lets you report both single-run success and repeated-run reliability. Fewer runs make flakiness invisible.

What latency numbers should I report?

Report P50, P95, and P99 from end of caller speech to audible response start, measured end to end. For cascaded pipelines, also break latency down by STT, LLM, and TTS stage so you can see which component threatens the real-time budget.

Conclusion

Benchmarking speech-to-speech models against cascaded STT, LLM, and TTS pipelines is only meaningful when the comparison is architectural: identical tasks, identical callers, deterministic scoring, repeated runs, a real-time latency budget, and paired clean and degraded audio. Under those conditions the two approaches reveal genuinely different strengths, and the results translate directly into a regression suite you can gate deploys with. Teams that want to run this kind of evaluation continuously, across voice, chat, and other modalities, can start with Bluejay.

If you want to run this comparison on your own agent, get started with Bluejay and test both architectures against your real call flows.