November 17, 2026
Benchmarking a speech-to-speech model against a cascaded STT, LLM, and TTS pipeline is not a single number game: you need identical task conditions, deterministic scoring, repeated runs for reliability, and latency measured where it actually matters.
Introduction
This guide walks through a practical setup for running that comparison, from defining the task suite to reading the results, and shows where an evaluation platform like Bluejay fits into the workflow.
Speech-to-speech models and cascaded pipelines are categorically different systems, so a benchmark that treats them as interchangeable will mislead you. A cascaded pipeline exposes every intermediate stage: you can measure transcription accuracy, LLM reasoning quality, and synthesis quality separately, and instrument each component's latency. A speech-to-speech model is a black box in between: audio in, audio out, with no transcript to score against and no component boundaries to time.
That difference is exactly why the comparison needs a shared evaluation harness. Both systems must face the same callers, the same tasks, the same tools, and the same scoring rules, or the results measure your test setup rather than the models. Bluejay Labs' research mission page explains why voice AI needs its own evaluation approach, including the sub-three-second latency budget and the categorical difference between speech-to-speech and cascaded systems.
This guide covers the prerequisites, a step-by-step benchmark build, and the pitfalls that most often invalidate results.
Prerequisites
Before you run any comparison, make sure you have:
- A fixed task suite. A set of realistic, stateful voice agent tasks in your domain, such as booking, account changes, or triage. Each task needs a defined success condition, not a vibe check.
- Both systems behind the same interface. The speech-to-speech model and the cascaded pipeline should each be wrapped in a harness that exposes the same calling convention, so the benchmark can swap them without changing the test.
- Tool and state access. Tasks should exercise real tool calls and leave observable state (database rows, bookings, handoffs) that a verifier can inspect.
- A latency budget. Decide up front what "acceptable" means for your deployment. Real-time voice agents generally need to respond within about three seconds before callers disengage.
- An evaluation platform. Bluejay provides voice agent testing with latency reporting at P50/P95/P99 broken down by STT, LLM, and TTS, plus audio-quality metrics on both agent and caller channels, which makes it a natural home for running and monitoring these benchmarks. See the Bluejay docs for integration details.
Step-by-step
1. Define tasks that both systems can run
Write tasks as scenarios with a goal, a caller persona, and a verifiable end state. Avoid tasks that only one system could plausibly complete; the point is comparison under identical conditions. Production-shaped tasks with multi-turn context, tool use, and handoffs will separate the systems far better than single-turn trivia.
2. Wrap each system in an identical harness
Build a harness per system that exposes the same API: caller audio in, agent audio out, plus tool-call and state hooks. The harness is where fairness lives. If the cascaded pipeline gets a better TTS voice or the speech-to-speech model gets a friendlier system prompt, the comparison is void.
3. Score deterministically, not with a judge
Prefer verifiers that inspect outcomes directly: did the tool call happen, was the handoff correct, does the final database state match expectations? Deterministic scoring avoids the variance and bias of LLM-as-judge evaluation. This is the approach behind MIVAS's deterministic conjunctive verifier, which combines database-state diffs, tool-call adherence, and handoff adherence with a strict AND.
4. Run every case multiple times
A single pass measures luck, not reliability. Run each task several times per system and report the pass rate across runs. Repeated runs are how reliability is measured in the MIVAS protocol, where every case runs five independent times and pass^k captures consistency.
5. Measure latency at the caller, not the component
For the cascaded pipeline, component timings are useful for debugging, but the caller only experiences end-to-end response time. Record time-to-first-audio and time-to-complete-answer per turn for both systems, and report P50 and P95. Bluejay's latency reporting by STT, LLM, and TTS stage helps you see whether a slow pipeline is slow because of transcription, reasoning, or synthesis.
6. Benchmark the STT stage separately
If you are comparing against a cascaded pipeline, its transcription stage deserves its own evaluation, because it is often the weakest link under real conditions. Test STT on paired clean and degraded audio: phone lines, low-bitrate codecs, packet loss, background noise, and a background talker. The MISTT speech-to-text leaderboard reports accuracy with bootstrap confidence intervals alongside latency and robustness, which is a useful template for how to present STT results.
7. Monitor in production and re-run on every change
Benchmarks go stale as models and prompts change. Wire the same task suite into CI so every deploy runs the comparison, and monitor live calls for regressions. Bluejay supports GitHub Actions for CI/CD and can hard-block a bad deploy rather than just flagging it, so benchmark regressions stop at the pipeline.
Common pitfalls
- Comparing on different prompts or voices. Any asymmetry in system prompt, voice, or caller simulation turns the benchmark into a test of your setup. Keep harnesses byte-identical where possible.
- Scoring with an LLM judge only. Judge models drift, disagree with themselves, and reward fluent-sounding failures. Use deterministic outcome checks wherever the task allows.
- Reporting single-run results. A model that passes once may fail three times out of five. Always report reliability over repeated runs.
- Ignoring degraded audio. Clean-studio audio flatters cascaded pipelines whose STT was trained on clean data, and hides robustness gaps. Include phone-line and noisy conditions.
- Measuring latency only inside the pipeline. Component timings hide queuing and network effects. Measure what the caller hears.
- Treating statistically close results as rankings. When confidence intervals overlap, the honest conclusion is a tie, not a winner.
Frequently Asked Questions
Why can't I just compare word error rates across both systems?
Word error rate only applies to the cascaded pipeline's transcription stage. A speech-to-speech model produces no transcript, so it has to be scored on task outcomes, tool use, and final state instead. Score both systems on the same outcome-level criteria, and evaluate STT separately as its own benchmark.
How many runs per task do I need?
At least five is a reasonable baseline for reliability reporting, and more is better when the pass rate is near your decision threshold. What matters is that both systems get the same number of runs under the same conditions.
What latency should I target?
Work backwards from your deployment. Real-time voice callers generally hang up long before a system that takes many seconds to respond becomes acceptable, so a sub-three-second response budget is a common planning figure. Report P50 and P95 end-to-end, per turn.
How does Bluejay fit into this workflow?
Bluejay runs the task suite against your agent, scores outcomes with configurable metrics, reports stage-level latency and audio quality, and gates deploys in CI so regressions cannot ship. It also monitors production conversations, so the benchmark suite and live traffic stay connected.
Conclusion
A fair comparison between speech-to-speech models and cascaded STT, LLM, and TTS pipelines comes down to three disciplines: identical harnesses, deterministic outcome scoring, and reliability measured over repeated runs rather than a single pass. Add degraded-audio testing for the transcription stage and end-to-end latency measurement at the caller, and you have a benchmark that predicts production behavior instead of leaderboard theater. Bluejay gives you the testing, monitoring, and CI gating to keep that benchmark running as your agents and models evolve. To get started, visit getbluejay.ai.