Bluejay × G2

Every voice AI platform demos well. The agent answers instantly, hears perfectly, books the appointment. Then you ask the question a buyer actually needs answered: if I build the same agent on six different platforms, which one completes a real caller’s request, against real business systems, on a bad phone line, five times out of five? Until now, nothing public answered that.

Today, together with G2, we are launching the Voice AI Agent Platform Evaluations: a public, measured comparison of the platforms teams build voice agents on. G2 chose Bluejay as its evaluation partner for voice, and the first run scores six platforms — Retell, ElevenLabs Agents, LiveKit Agents, Pipecat, Vapi, and Bland — with every number, chart, and methodology decision public on G2.

Reviews meet measurements

G2 is where software buyers go to compare products, and its reviews capture something a benchmark never will: what it is like to live with a product. But AI agents add a question reviews cannot settle. A voice agent platform is a runtime that answers the call, decides when the caller has finished speaking, executes tool calls, and wires speech-to-text, an LLM, and text-to-speech into one conversation. How well it does that is measurable, and buyers deserve the measurement next to the reviews.

That is what G2’s Agent Evaluations program is for, and voice is the category where the gap between a demo and production is widest. Bluejay runs the voice category end to end: the agent, the simulated callers, the grading, and the re-runs. The task suite and grading derive from MIVAS, Bluejay Labs’ benchmark of multi-step, tool-using voice agents, and where a platform leaves model choice open, the speech-to-text pick comes from MISTT, our speech-to-text bench. More than 5,000 simulated calls went into building and running this first version.

How the evaluation works

The unit of evaluation is the platform, so the agent is held constant. Every platform hosts the same system prompt, greeting, and tool schemas, deployed byte for byte, and takes the same calls. A difference in outcome is a difference in how platforms run an agent.

The calls come from three simulated businesses, each modeled on a real deployment and the rules that govern that kind of business: Straus Dermatology, a healthcare front desk with 160 locations and 380 providers, where identity must be verified before any chart or balance is disclosed; Halverson & Reed, a plaintiff-side injury firm whose intake line screens for conflicts before any facts are heard; and Kestrel Electronics, a retailer with about 1,000 stores, membership tiers, and state-dependent restocking fees. The agents carry production-length instructions — 2,131 to 3,523 words of system prompt — and real tool servers with seeded databases behind them.

Each platform takes 630 simulated calls: 90 tasks run five times each, plus 36 copies of easy tasks under background noise or a degraded phone line, also run five times. The callers are Bluejay digital humans at temperature 0, with every fact the grade depends on fixed by scripted responses, so each task reaches every platform in the same words, in the same voice, on every run.

Grading is deterministic. A call passes only if the tool server’s log shows every required tool called with the right arguments and the business’s database ends in the expected state. An agent that says the right thing but writes the wrong record fails; so does one that reaches the right state by skipping an identity check. No LLM judge, no partial credit. The platform score is out of 100: up to 80 points for task success, up to 10 for how accurately the platform transcribed the caller on clean and degraded audio, and up to 10 for latency, measured on our own stereo recordings from the caller’s last word to the agent’s first audio.

Fairness is structural, not aspirational. Calls broken by our side are voided, fixed, and re-run on every platform. Where Bluejay chose the models for an open framework (LiveKit Agents and Pipecat run GPT-6 Sol, Cartesia Ink-2, and ElevenLabs Eleven v4 Turbo, selected by a fixed published rule), failures caused by those models are excluded from the task score and published separately, so an open framework is not penalized for a model we chose and a closed platform is not credited for a model it did. The full methodology, down to every formula and failure-attribution rule, is public on G2.

What the first run says

Here is the leaderboard from the first public run:

First public run, October 2026 (methodology v0.4, manifest v5). Task success is pass@1 on the 450 clean scored calls per platform. The live leaderboard, with per-industry views, Pareto frontiers, reliability charts, and platform profiles, is at ai.g2.com.

Three findings stand out.

First, the top is a three-way band, and each platform buys its score differently. Retell (81.4), ElevenLabs Agents (80.9), and LiveKit Agents (80.4) finish within one point of each other, and all three sit on the Pareto frontier: no platform is both faster and more successful at tasks than any of them. But LiveKit Agents posts the highest task success in the field (89.1%) while making callers wait longest (3.30 s median turn); ElevenLabs Agents answers almost three times faster (1.15 s) and completes fewer tasks (81.1%); Retell splits the difference. Picking a platform at the top of this leaderboard is not picking a winner, it is picking a trade-off.

Second, the gap between a demo and production is measurable, and it is large. Because every task runs five times, the evaluation reports two numbers: pass@1, the share of single calls that pass — what a demo shows — and pass^5, the share of tasks that pass on all five runs — what a caller can count on. The top three platforms resolve 81–89% of single calls, but only 47–55% of tasks survive all five runs. LiveKit Agents has the widest gap: 89.1% of its single calls pass, and 47.2% of its tasks pass every time, 41.9 points apart. Run-to-run consistency, not the highlight reel, is where voice AI still breaks.

Third, the industry reorders the leaderboard. Pipecat posts the single best number in the run — 98.0% task success on retail customer support — and drops to 71.3% on legal intake, where conflict screening and admissibility gates punish an agent that overreaches. LiveKit Agents leads legal at 82.7%; Pipecat leads healthcare at 88.7%. The same agent, graded the same way, ranks differently depending on whose phone line it answers, which is exactly why the leaderboard ships with per-industry views.

What buyers should do with this

If you are choosing a platform, start from the workload, not the headline score. The per-industry tables say more about your deployment than the average does, the Pareto charts show which platforms are dominated outright, and the reliability chart tells you how much re-run consistency you are giving up for demo-day speed. Beside the score, each platform also gets a qualitative profile — UI, developer experience, docs, go-live path, pricing clarity, language support — graded on cited evidence and kept out of the score itself.

If you build one of these platforms, the evaluation is a target shaped like production: policy gates, two-phase writes, callers that refuse the second save offer, audio that sounds like a parking lot. Everything is versioned in a frozen manifest, and any change re-runs every platform, so the leaderboard can only be moved by actually getting better.

The evaluation comes from the same place as the product. The digital humans placing these calls, the degraded phone lines, and the tool-level grading are what Bluejay customers already use to test their own agents before and after they ship. The partnership with G2 puts that measurement where buying decisions actually happen.

Explore the evaluation

The leaderboard, per-industry results, Pareto frontiers, reliability charts, and the full methodology are live on G2.