The language model at the center of a voice agent has a strange job description. It reads a 2,500-token policy, verifies a caller’s identity, queries the right system, quotes the right fee, and writes the right record. And it must start answering in under a second, because a caller on a phone line is sitting in silence. Pick that model by reasoning benchmarks alone and it may be too slow to put on a phone line. Pick it by the speed-and-price pages, and nothing tells you whether it books the appointment before verifying the patient, or writes a misheard date of birth into the chart.
Today we are launching CALLM, the Conversational AI LLM Bench: a public benchmark from Bluejay Labs that defines this category of model and measures it. The first run scores 29 models on 180 locked tasks across healthcare, legal, and customer support, 26,100 scored conversations in all, with every number public at research.getbluejay.ai.
What is a conversational AI LLM?
Most phone agents in production are cascaded systems: speech recognition turns the caller’s audio into text, a language model decides what to say, which tool to call, and when to hand the conversation to another agent, and speech synthesis speaks the reply. Recognition and synthesis carry the words. The model in the middle is the only stage that reads the business’s policy, calls the business’s systems, and writes to its records.
A conversational AI LLM (CALLM) is a language model that can hold that seat: it sits between speech recognition and speech synthesis and returns its first token within one second, a median time to first token of 1,000 ms or less. Speed is part of the definition because the caller waits in silence until the first token arrives. People answer each other within a few hundred milliseconds, the recognizer and synthesizer add their own delay on top of the model’s, and synthesis streams, so starting matters far more than finishing. A model that needs several seconds to begin is not usable on a phone line, however accurate it is, and ranking it beside models that are would compare two different products.
The line has teeth. GPT-6.1 Sol measures 1,405 ms, so it is not ranked; it plays the simulated caller instead. Amazon’s Nova 2 Lite misses at 1,136 ms. NVIDIA’s Nemotron 3 Nano is listed with reasoning off (223 ms) and excluded with it on (3,014 ms). Every model is measured again with each release, so membership in the category can change.
Why the LLM stage needed its own bench
Several vendors now ship language models trained specifically for phone and voice agents. Their published comparisons report speed and price. We know of no evaluation that compares them on the same tasks, tools, and caller, either against each other or against the frontier and open-weights models they compete with.
General LLM benchmarks evaluate knowledge or reasoning in a single prompt. A deployed phone agent fails differently. It books an appointment before the caller’s identity is verified. It answers a coverage question from its prior instead of calling the eligibility tool. It agrees to a refund that policy forbids, or writes a misheard date of birth into the patient record. These failures are invisible in a transcript that reads well, and perfectly visible in the tool trace and the final database.
CALLM is Bluejay Labs’ third public benchmark, and it completes a set. MIVAS measures whole voice systems end to end. MISTT isolates the ears: speech-to-text under the audio real calls are made of. CALLM isolates the brain, and asks the two questions that decide a deployment: does the model complete the task as the business requires, and does it do so on every attempt?
What is CALLM?
CALLM runs each model as the intelligence layer of three fictional companies, in sectors where conversational agents are already deployed: Straus Dermatology (a healthcare front desk with seven specialist agents), Halverson & Reed (plaintiff-firm legal intake under conflict-before-facts rules), and Kestrel Electronics (retail support behind an order-bound identity gate). Each company is a directed graph of specialist agents with production-length prompts (the median agent prompt runs 2,500 to 3,200 tokens), executable tools, and a seeded SQLite database that every call gets a private copy of. The model under test runs every agent: reception, identity, scheduling, coverage, billing, fraud. A fixed caller model pursues a locked task, and the call ends when either side hangs up or a 30-exchange cap is reached.
The suite is 180 tasks, 60 per industry, with the same composition everywhere: six categories of ten, each with two easy, four medium, and four hard tasks. Five categories cover the company’s main workflows; the sixth covers regulation, refusals, escalation, impersonation, and adversarial requests. Every task locks the caller’s brief and traits, the required tool calls with their arguments, the expected handoff path, and the final database state. The full suite is held out to limit contamination; the agent prompts and nine sample tasks are public.
How CALLM works
Three design choices define the bench, and each exists because of how phone agents fail in production.
The model hears what a recognizer transcribed, not what the caller said
A phone agent never receives the caller’s words; it receives what a speech recognizer made of them. CALLM reproduces that gap in text. A simulated phone line sorts each caller turn by kind and copies the whole set of errors from a real turn of the same kind, drawn from 4,769 caller turns in our voice-agent benchmark calls where we have both the caller’s scripted text and what the recognizer heard over a phone line. Errors happen as often, and as often together, as on real calls, and draws are seeded so attempt k of a task meets the same errors for every model. To check the channel, a judge model shown one real and one simulated transcript of the same call picked the real one 58% of the time, where 50% would mean the two cannot be told apart.
Deterministic verification, conjunctive like MIVAS
After each call, three checks run, with no LLM judge anywhere: the required tool calls fired with the locked arguments (tool adherence), the expected specialist transfers appeared in order (handoff adherence), and the final database matches the expected state (database adherence). A call passes only if all three pass, because no single check is sufficient. One healthcare task asks whether an office accepts Cigna before booking: the eligibility check is a read and leaves no row behind, so a model that skips it and books correctly reaches exactly the expected database, and only the tool check catches the omission. And a call can make every required call and still write a wrong value: a name the model wrote down as misheard fails the database check.
pass@1 measures capability; pass⁵ measures reliability
Every task is attempted five times. pass@1 is the share of single attempts that pass; pass⁵ is the probability that all five attempts of a task pass, so a task contributes only if it never failed. The leaderboard ranks by pass⁵, because a deployment handles the same request many times, and a model that completes a task on three of five attempts will fail a predictable share of real callers. Latency and cost are reported alongside the scores and do not affect pass or fail; latency decides only whether a model is admitted to the category at all.
What the first scores say
All 29 models ran the frozen suites in November 2026. Here is the top of the cross-industry table:
| # | Model | Provider | pass⁵ | pass@1 | Latency | Cost/hr |
|---|---|---|---|---|---|---|
| 1 | Claude Haiku 5.5 | Anthropic | 62.2% | 83.2% | 963 ms | $0.206 |
| 2 | Tell-1 | Retell AI | 60.6% | 80.0% | 352 ms | N/A |
| 3 | DeepSeek V4.1 Flash | DeepSeek | 57.8% | 81.1% | 476 ms | $0.031 |
| 4 | Gemma 4 31B | 46.7% | 71.5% | 383 ms | $0.306 | |
| 5 | PhoneLLM Max Alpha 2 (microthinking) | Pipecat | 42.8% | 67.8% | 499 ms | $0.222 |
| 6 | GPT-4.1 | OpenAI | 41.7% | 71.3% | 693 ms | $0.894 |
| 7 | GPT-4.1 mini | OpenAI | 40.0% | 66.8% | 653 ms | $0.245 |
| 8 | Qwen3.5 397B A17B | Alibaba | 39.4% | 68.8% | 395 ms | $0.234 |
| 9 | Gemini 2.5 Flash | 38.9% | 66.0% | 610 ms | $0.132 | |
| 10 | Claude Haiku 4.5 | Anthropic | 36.7% | 65.0% | 459 ms | $1.34 |
Top 10 of 29 by cross-industry pass⁵, November 2026 run: the unweighted mean of healthcare, legal, and customer support. Latency is median time to first token; cost applies list token prices over estimated call hours. Retell has not published a price for Tell-1. The live table, with per-industry splits, difficulty breakdowns, per-check adherence, and all 29 models, is at research.getbluejay.ai.
Three findings stand out.
First, the podium is three different bets. Anthropic’s Claude Haiku 5.5, a general frontier small model, leads at 62.2% pass⁵ and squeezes under the admission line at 963 ms, with 37 ms to spare. Retell’s Tell-1, a model trained specifically for phone agents and released this week, is second at 60.6% and answers nearly three times faster, at 352 ms. DeepSeek V4.1 Flash, open weights, is third at 57.8% for $0.031 an hour: about three cents of tokens per hour of conversation, roughly a seventh the cost of the leader. A frontier lab, a voice-agent vendor, and open weights: no one strategy owns the category yet.
Second, the gap between a good demo and a reliable agent is the widest number on the board. The leader passes 83.2% of single attempts but only 62.2% of tasks five-for-five, and the gap widens down the table: GPT-5 mini passes 37.1% of attempts but only 12.8% of tasks reliably. Difficulty cuts the same way; Claude Haiku 5.5 goes from 96.7% pass@1 on easy tasks to 75.8% on hard ones, and every model drops. A model chosen on a one-shot demo call will look much better than it behaves at volume, which is exactly why the leaderboard ranks by pass⁵.
Third, the frontiers are nearly empty. On latency versus accuracy, only six of 29 models are undominated, and between the ends of that staircase the trade is 740 ms of median latency for 54.4 points of pass⁵. On cost versus accuracy, among the 27 models with published prices, the frontier is just two: DeepSeek V4.1 Flash and Claude Haiku 5.5. Everything else is slower, less reliable, or more expensive than something above it, including most of the models trained specifically for this job. Purpose-built helps (Tell-1 is second, Pipecat’s PhoneLLM Max Alpha 2 is fifth), but it is a bet, not a guarantee: other phone-trained models land in the bottom third of the table.
CALLM states its limits plainly. The caller is a language model following a brief, more consistent and more cooperative than real callers. The channel is text-level: it reproduces measured transcription error rates but not timing, interruptions, or prosody, and it leaves out the one error no agent could catch without a read-back, a clean swap of one digit for another. Three fictional companies do not represent every deployment. And five attempts per task make each per-task pass⁵ estimate binary, so differences of a few points should be read alongside the bootstrap intervals the full release reports.
Why public scores matter
Teams shipping voice agents currently pick the model in the middle from speed-and-price pages, while model providers optimize for the benchmarks that exist. Until now, none of those benchmarks looked like a phone call with a database behind it. CALLM gives buyers evidence and gives providers a target shaped like the job.
The bench comes from the same place as the product. Bluejay simulations exercise production voice agents with simulated callers over degraded lines, because that is where agents fail first; CALLM points the same machinery at the models themselves. The environment is packaged for NeMo Gym, so the code that produces the leaderboard also runs as a reinforcement-learning environment, and all of it is in the open repository: environments, verifier, reporting code, and sample tasks.
This is a working paper, and the category it names is moving fast: new phone-trained models keep shipping, and every release gets measured again. CALLM will track whether the models at the center of voice AI are actually getting better.
Explore the CALLM Bench
The full leaderboard, methodology, industry packs, and sample tasks are live on the research site, and the evaluation environment is on GitHub.


