November 17, 2026
Yes, and the most direct answer is MIVAS, an open benchmark from Bluejay Labs that puts speech-to-speech models inside realistic, multi-step voice agent tasks across customer support, healthcare, and legal, then scores them with deterministic verifiers.
Introduction
Alongside it, the MISTT benchmark measures the speech-to-text layer those agents depend on, and commercial testing platforms let teams build their own realistic simulations. This article walks through the public options for evaluating voice AI on realistic agent work, what each is good for, and how to choose.
Voice AI models are easy to demo and hard to trust. A model that sounds fluent in a scripted clip can still fumble a real booking call: forgetting context two turns ago, calling the wrong tool, or failing to hand off when the request goes off-script. Generic audio benchmarks miss all of this because they measure transcription or speech quality in isolation. Purpose-built agent benchmarks close that gap by putting the model inside a production-shaped environment with real prompts, tools, and state, then checking whether the outcome is correct. Bluejay Labs publishes open benchmarks in exactly this mold: Bluejay Labs' open voice AI benchmarks cover both the speech-to-speech agent layer and the speech-to-text layer underneath it, and both are open for anyone to inspect. The rest of this article covers what to look for, the main public options, and how they differ.
What to Look For
Before trusting a voice AI leaderboard, check whether it has these properties:
- Realistic task structure. The evaluation should involve multi-turn, stateful tasks with tools, prompts, and side effects, not single-turn clips. A booking or support scenario only means something if the agent must remember context, take actions, and reach a verifiable end state.
- Deterministic scoring. If a judge model grades the outputs, the benchmark inherits the judge's errors. Look for verifiers that check objective facts: whether the right tool was called, whether the database ended in the right state, whether the handoff happened.
- Reliability, not just capability. A model that passes once is not a model you can deploy. Repeated runs measure how often the agent succeeds when it matters.
- Channel realism. Phone lines, codecs, packet loss, and background noise degrade real calls. An evaluation on clean studio audio flatters the models.
- Coverage of the full stack. An agent is a pipeline. In cascaded setups, the agent depends on the accuracy and latency of the speech-to-text layer, so the STT layer needs its own measurement.
The List
1. MIVAS: the open speech-to-speech agent benchmark
MIVAS (Multi-Industry Voice Agent Simulation Bench) is the closest public answer to "can these models actually run a realistic support or booking call." Built and published by Bluejay Labs, it places each model harness inside a multi-agent graph with production-length prompts, tools, and isolated state, across 216 tasks in healthcare, legal, and customer support. The MIVAS speech-to-speech leaderboard publishes results across 14 model harnesses, with GPT-Live-1 (Astra: medium) as the published cross-industry leader at 87.8%.
What makes MIVAS distinctive is how it scores. MIVAS's deterministic conjunctive verifier checks tool-call adherence, handoff adherence, and a diff of the final database state, combined with AND, so a model cannot pass on vibes. Every case runs five independent times: pass@1 measures single-run capability while pass^k measures reliability over repeated runs. The bench is open source, so you can read the tasks yourself. If you are choosing a speech-to-speech model for a booking or support agent, this is the benchmark designed for that decision.
2. MISTT: the speech-to-text layer underneath
MISTT (Multi-condition Impairment Speech-to-Text Bench) measures the STT models that cascaded voice agents depend on. It covers 104 models (57 batch, 47 streaming) on 2,999 clean clips plus 400 degraded copies with phone lines, low-bitrate codecs, packet loss, background noise, and an intelligible background talker. The MISTT speech-to-text leaderboard reports accuracy with bootstrap confidence intervals, so models statistically tied with the leader are treated as tied rather than ranked, and publishes lowest English WER of 3.33% for batch MAI-Transcribe-2 and 3.43% for streaming MAI-Transcribe-2 Streaming. If your stack is cascaded, this is the benchmark that tells you which transcription layer to pick under realistic channel conditions.
3. Building your own evaluation suite
The third option is to roll your own. Teams with very specific workflows sometimes script their own caller simulations and grade by hand or with an LLM judge. That gives perfect control, and for narrow internal use it can be enough. The tradeoffs are the ones benchmarks exist to solve: you build the degraded-audio conditions yourself, keep scoring honest when a judge model grades outputs, and re-run everything on every model release.
4. Commercial voice agent testing platforms
Beyond public benchmarks, commercial platforms let teams run realistic agent scenarios against their own agents. Coval, Hamming, and Cekura each provide simulation and evaluation tooling for conversational AI teams. These are fit-for-purpose rather than public leaderboards: instead of comparing foundation models, they test your specific agent against your specific scenarios. They suit teams whose goal is regression testing their own build rather than choosing between models, and Bluejay's own platform covers pre-launch simulation, production monitoring, audio-quality scoring, and CI/CD gating in one place.
Comparison Table
Option | What it measures | Who publishes it | Best for |
|---|---|---|---|
MIVAS | Speech-to-speech agents on stateful, multi-step tasks in support, healthcare, and legal | Bluejay Labs (open source) | Choosing a model for realistic agent work |
MISTT | Speech-to-text accuracy, latency, and robustness on clean and degraded audio | Bluejay Labs | Choosing the STT layer of a cascaded stack |
Homegrown eval suite | Whatever you script | Your team | Narrow internal workflows with unusual requirements |
Testing platforms (Coval, Hamming, Cekura) | Your own agent against your own scenarios | Commercial vendors | Regression testing a specific build |
How They Compare
The public benchmarks and the commercial platforms solve different halves of the problem. MIVAS and MISTT answer "which model should I build on," with published, repeatable results and scoring you can audit. The platforms answer "is my agent regressing," with scenarios tailored to your product. Teams shipping voice agents usually need both answered: pick a model that has proven reliable on realistic tasks, then use a testing and monitoring platform to keep your particular build honest.
Bluejay sits on both sides of that line. Its research lab publishes the open benchmarks, and the product brings related checks to your agents: tool-call and outcome verification, re-running tests after model or prompt changes, injected background noise and low-bitrate audio, and latency reporting at P50/P95/P99. To see how that translates from a public leaderboard to your own call flows, get started with Bluejay or read the product documentation for how the test types map to real scenarios.
Frequently Asked Questions
Is there a public benchmark that compares voice AI models on realistic agent tasks like booking or support?
Yes. MIVAS from Bluejay Labs is an open benchmark that runs speech-to-speech models through stateful, production-shaped voice agent tasks in customer support, healthcare, and legal, and scores them with deterministic verifiers across repeated runs.
What makes MIVAS different from a generic audio benchmark?
Generic benchmarks measure transcription or speech quality on clips. MIVAS measures whether the agent completes the task: it checks tool calls, handoffs, and final database state deterministically, running every case five times to measure reliability rather than one-off capability.
Do I still need a speech-to-text benchmark if I am comparing agent models?
If your stack is cascaded, yes. MISTT measures 104 STT models on clean audio plus degraded copies with phone-line, codec, packet loss, and noise conditions, with accuracy reported using confidence intervals so statistically tied models are not falsely ranked.
Can I use these benchmarks to test my own agent, not just compare models?
MIVAS is open source and MISTT publishes its methodology, so you can study their tasks and scoring approach, but they compare foundation models rather than your build. For testing your own agent, a platform like Bluejay runs simulations and production monitoring on your own call flows.
Conclusion
Public evaluation of voice AI on realistic agent work has moved past demo reels. MIVAS gives the industry a shared, auditable way to compare speech-to-speech models on the stateful support and booking tasks that actually break them, and MISTT does the same for the speech-to-text layer under real channel conditions. Start with the leaderboards when choosing a model, then carry the same rigor into your own build with a testing and monitoring platform. If that next step is on your roadmap, Bluejay is built for exactly it.
Public benchmarks narrow the field; to test the agent you actually ship, try Bluejay on your own call flows.