Introducing MISTT

Ask any speech-to-text leaderboard and the problem looks solved: the best models transcribe benchmark audio with word error rates a careful human would envy. Then the same model takes a real call: a cell phone on speaker, a G.711 phone line, a codec squeezing the voice to 8 kbps, packets dropping, a television in the background, a second person talking over the caller. The transcript every downstream decision depends on quietly falls apart.

Today we are launching MISTT, the Multi-condition Impairment Speech-to-Text Bench: a public benchmark from Bluejay Labs that scores the speech-to-text models voice agents depend on, under the conditions those agents actually hear. The v0 run covers 81 models from 25 vendors (45 batch, 36 streaming) on one frozen test set, with every score public at research.getbluejay.ai.

Why the STT stage needed its own bench

Most production voice agents are cascaded systems: speech-to-text in front, a reasoning model in the middle, text-to-speech on the way out. Which tool to call, which number to read back, whether the caller said fifteen or fifty: everything the agent decides is decided over the transcript. If the transcript is wrong, no prompt downstream can fix it.

Yet STT models are bought on clean-audio numbers. Vendors quote word error rates from studio-quality recordings, and the public benchmarks that exist mostly reward the same thing. Production calls are not studio audio. Real-world audio is rarely pristine, and a benchmark that only scores pristine audio cannot tell you which model survives contact with a phone line.

MISTT is Bluejay Labs’ second public benchmark. MIVAS measures whole voice systems end to end: tools, handoffs, final database state. MISTT isolates the stage those systems lean on hardest, and asks the question MIVAS cannot: when the audio degrades, which transcription survives?

What is MISTT?

MISTT (Multi-condition Impairment Speech-to-Text Bench) measures the speech-to-text models that voice agents depend on. Every model hears the same 2,999 frozen clips, plus 400 degraded copies of 100 of them. The clips span 1,000 voice-agent caller turns from pipecat’s stt-benchmark-data, 1,000 doctor-patient consultation turns from primock57, and 999 clips of five-language parliamentary speech from VoxPopuli.

The headline metric is English WER: an equal-weight mix of voice-agent caller turns and doctor-patient turns, fixed before any result was seen. Every accuracy figure carries a 95% clustered bootstrap confidence interval, and models whose interval overlaps the leader’s are shaded as a band. MISTT does not pretend to rank within it.

How MISTT works

Three design choices define the bench, and each one exists because of how voice agents fail in production.

One frozen test set, one run per model

Every model version transcribes the same clips exactly once, from the same host, at pay-as-you-go rates. The reported $ per 1,000 minutes is the effective cost of the run (billing increments, minimums, and silence tails included), not the rate card number.

Robustness, measured on the same words

The degraded copies come from four impairment mixes: Light (a phone-line band-pass and G.711 μ-law codec), Moderate (background noise at 15 dB SNR, 8 kbps Opus, 2% bursty packet loss), Severe (noise at 10 dB SNR, a phone line, 5% packet loss), and Crosstalk (an intelligible background talker at 10 dB, then a phone line). Each degraded clip is paired with its clean original, so the change in WER reflects the same words: the drop is the model’s, not the material’s. Crosstalk gets one more metric: bleed, the share of the background talker’s words that end up in the transcript.

Latency, measured the way an agent waits

Streaming models receive audio in real time as 20 ms frames. When voice activity detection marks the end of speech, the harness sends a finalize message, and the clock runs from that forced end of speech until the final transcript arrives. That is the silence a caller sits through before the agent can start thinking, which makes it the only latency number an agent team can actually budget around.

What the first scores say

All 81 models ran the frozen v0 suites in October 2026: 275,319 scored transcriptions in total. Here is the top of the English WER table:

Top 10 of 81 by English WER, v0 October 2026 run. * free, rate-limited endpoint. ~ token-priced estimate. † measured via a fallback; not directly comparable. The live table, with confidence intervals, per-dataset splits, robustness deltas, and all 81 models, is at research.getbluejay.ai.

Three findings stand out.

First, the top is a band, not a podium. Pulse Pro, an English-only batch model from Smallest.ai, posts the lowest English WER at 3.38%, but eleven batch and fifteen streaming models land inside their leaders’ confidence intervals, and MISTT refuses to rank within that band. At the top of this market, choosing by headline WER alone is reading noise; the real differences are latency, cost, language coverage, and what happens when the audio degrades.

Second, impairments reorder the leaderboard. Under the Light mix the clean ranking mostly holds (Kendall τ 0.703 against clean); under Crosstalk it dissolves (τ 0.374), and the median model gives up 11.7 WER points. The sharpest case is ElevenLabs’ Scribe v2: first on the robustness probe’s clean clips, 79th under Crosstalk, transcribing roughly a quarter of the background talker’s words into the transcript. Zoom Scribe EN Pro held up best under the Severe mix, losing just 2.0 points.

Third, the audio you serve matters more than the model you pick. The field’s median WER is 2.40% on voice-agent caller turns and 6.91% on doctor-patient consultations: same models, harder audio, harder vocabulary. And streaming has a real price: in same-vendor pairs, streaming costs a median 0.36 WER points over batch. That is the tax for answers that arrive in milliseconds (Cartesia’s Ink-2 returns its final transcript 73 ms after the caller stops talking) instead of seconds.

MISTT v0 also states its limits plainly. The three base datasets are public, so clean scores may flatter models that saw them in training. The degraded copies are new audio, which is part of why they matter. All requests ran from a single AWS us-west-2 host, so vendors hosted nearby get a network advantage on latency. And entity accuracy, the digits, amounts, and account numbers that cost real money when they flip, is designed but deferred to a future version.

Why public scores matter

Teams shipping voice agents currently pick their STT from vendor demo pages and clean-audio WERs. Model providers, in turn, optimize for the benchmarks that exist, and until now none of them scored transcription under the conditions a deployed agent actually hears. MISTT gives buyers evidence and gives providers a target shaped like a real phone call.

The bench comes from the same place as the product. Bluejay simulations already place test calls under background noise and degraded lines, because that is where production agents fail first. The lab and the platform share one thesis: you cannot make voice AI trustworthy without measuring it under the conditions it will actually face.

This is a v0 working paper, and it will grow: entity-level accuracy, more crosstalk variants, more languages, and new model versions as vendors ship them, with public scores tracking whether the field is actually getting better.

Explore the MISTT Bench

The full leaderboard, methodology, datasets, and the white paper are live on the research site.