November 17, 2026
A public voice AI benchmark is an openly published evaluation that runs speech models and voice agents through realistic tasks, such as booking an appointment or resolving a support call, and scores the results against a fixed methodology.
Introduction
The best ones go beyond transcription accuracy and measure whether an agent completes the job: calling the right tools, following the right handoffs, and leaving the system in the right final state.
If you are choosing a speech-to-speech model or a speech-to-text provider for a voice agent, most published numbers tell you very little about what you actually care about. A model can post a strong score on a generic audio clip and still fall apart on a real phone call, where the caller talks over it, the line degrades the audio, and the agent has to juggle tools, prompts, and state across a multi-minute conversation.
That gap is why public benchmarks built around realistic agent tasks matter. Instead of asking "how well does this model transcribe studio audio?", they ask "can this model run a booking flow end to end, five times in a row, on a noisy line?" This article explains what public voice AI benchmarks exist, what they measure, how to read their leaderboards, and where their limits are.
Key Takeaways
- Public benchmarks for voice AI do exist, and the best evaluate agents on realistic tasks like booking, scheduling, and customer support rather than isolated audio clips.
- Task realism comes from production-shaped scenarios, tool and state verification, and repeated runs so reliability is measured, not assumed.
- Deterministic scoring, where the benchmark checks final database state and tool calls directly, avoids the variability of an AI judge model.
- Speech-to-text benchmarks matter too, because a voice agent is only as good as the transcript it reasons over, especially on phone lines and noisy audio.
- Benchmarks are a shortlist tool, not a substitute for testing a model inside your own agent before you ship.
What Counts as a Realistic Agent Task
A realistic agent task resembles the calls your agent will actually take: the caller's goal may shift mid-conversation, the agent has tools it must call in the right order, and the interaction ends in a verifiable outcome such as an appointment created or a ticket escalated.
Generic speech benchmarks usually stop at transcription or a single-turn response. Agent-level benchmarks go further: they place the model inside a simulated environment with production-length prompts, tools, and isolated state, then check whether the agent did the job. The MIVAS task catalog shows what this looks like in practice: stateful, multi-step scenarios drawn from healthcare, legal, and customer support, where the agent has to manage tools and handoffs rather than just answer a question.
The industries matter as much as the tasks. A scheduling call at a medical clinic carries different compliance and edge-case risks than a billing dispute in customer support, so a benchmark that spans industries shows where a model is strong and where it is brittle.
How Public Benchmarks Score Voice Agents
The scoring method is where serious benchmarks separate themselves from marketing exercises. Two design choices stand out:
First, deterministic verification. Instead of asking an AI judge "did the agent do a good job?", a well-designed benchmark checks concrete facts: did the agent call the right tools, follow the handoff rules, and leave the final database state as the scenario required? MIVAS's deterministic conjunctive verifier combines these checks with a logical AND, so an agent fails if any required condition fails, removing judge-model bias and making scores reproducible.
Second, repeated runs. A single pass tells you what a model can do on its best day. Reliability is better judged over repeated runs, which is how the MIVAS evaluation protocol scores every case: each scenario runs five independent times, and the benchmark reports both pass@1 (single-run capability) and pass^k (reliability across repeated runs). For a production agent, that second number often matters more, because a flow that works four times out of five frustrates one caller in five.
Where to Find the Leaderboards
Bluejay Labs publishes open benchmarks at research.getbluejay.ai, covering both sides of a voice agent's stack.
For speech-to-speech models, the MIVAS speech-to-speech leaderboard ranks model harnesses on those realistic, stateful agent tasks across healthcare, legal, and customer support. It covers 14 model harnesses across 216 tasks, including both realtime speech-to-speech models and cascaded pipelines, which matters because the two architectures behave very differently under a latency budget and should not be compared as if they were interchangeable.
For the transcription layer underneath, the MISTT speech-to-text leaderboard evaluates 104 models on clean audio plus degraded copies that simulate phone lines, low-bitrate codecs, packet loss, background noise, and a background talker. Accuracy is reported with bootstrap confidence intervals, and models whose interval overlaps the leader's are treated as statistically tied rather than ranked below it, which prevents readers from drawing conclusions the data cannot support.
MIVAS is open source and MISTT publishes its methodology and reproduction details, so you can inspect the harnesses and methodology rather than trust a black-box score.
How to Read a Voice AI Benchmark Without Getting Misled
Public benchmarks reward careful reading. A few habits keep you honest:
- Look at reliability, not just peak performance. A high pass@1 with a much lower pass^k means the model can succeed but not consistently.
- Check whether the tasks resemble your calls. A benchmark strong on customer support scenarios may say little about legal intake or clinical scheduling; use the industry breakdowns.
- Prefer deterministic scores over judge models. If the benchmark's verdict comes from another LLM, the score inherits that model's biases and variance.
- Watch the audio conditions. Models ranked on clean studio audio may reorder badly on real telephony; paired clean-and-degraded testing is the standard to look for.
- Treat close scores as ties. When a leaderboard reports confidence intervals, models inside the leader's interval are tied, not "almost first."
The Limits of Public Benchmarks
Even the best public benchmark evaluates someone else's scenarios. Your agent has your prompts, your tools, your knowledge base, and your callers. A model that leads a leaderboard can still fail on your flows, and a mid-pack model can excel at exactly your use case.
So treat benchmarks as a shortlist generator: use them to narrow dozens of candidates to two or three that perform well on tasks resembling yours, then test those finalists inside your own agent before committing. That is where a platform like Bluejay fits: it runs automated simulations of your real call flows, scores them with LLM-as-a-judge, ML-model, and statistical metrics, and gates regressions in CI/CD so a model swap or prompt change cannot silently break production. The testing and monitoring workflow is documented in Bluejay's docs.
The combination is what actually de-risks a launch: public benchmarks for the model shortlist, your own simulations for the ship decision.
Frequently Asked Questions
Are there public benchmarks that test voice AI on realistic agent tasks like booking or support calls?
Yes. MIVAS, published openly by Bluejay Labs, places speech-to-speech model harnesses inside stateful, multi-step scenarios in healthcare, legal, and customer support, with production-length prompts, tools, and isolated state, scored deterministically over five runs per case.
Why do some benchmarks run every task multiple times?
Because a single successful run shows capability, not reliability. Running each case five times and reporting pass^k reveals how consistently a model completes the task, which is what production callers experience.
What is the difference between speech-to-speech and speech-to-text benchmarks?
Speech-to-speech benchmarks like MIVAS evaluate the whole agent loop: understanding, reasoning, tool use, and handoffs. Speech-to-text benchmarks like MISTT evaluate the transcription layer alone, including how accuracy holds up on phone lines, noise, and packet loss, and how fast streaming models produce a final transcript.
Can I rely on a benchmark score alone when choosing a model for my voice agent?
No. Benchmarks are best used to shortlist models that perform well on tasks similar to yours. Final validation should happen inside your own agent, on your own call flows, ideally as automated simulations wired into your deployment pipeline.
Conclusion
Public benchmarks that test voice AI on realistic agent tasks do exist, and they have matured quickly. The strongest ones, like MIVAS for speech-to-speech agents and MISTT for the speech-to-text layer, score agents on stateful, production-shaped scenarios with deterministic verification and repeated runs, and publish everything openly so you can check the methodology yourself.
Use them for what they are good at: narrowing a crowded model landscape to a defensible shortlist, grounded in tasks that look like booking, scheduling, and support rather than studio recordings. Then close the last mile with your own testing: run the finalists through simulations of your real call flows, monitor them in production, and gate deploys on regression tests. If you want a platform that handles that second half, Bluejay was built for exactly it.
When a leaderboard has done its job, set up Bluejay to measure your own agent on the tasks your callers bring.