Every conversation that touches Bluejay ends the same way: it gets graded. A simulation grades every conversation against its success criteria, custom metrics, tool call metrics and more. Production monitoring does the same for every production conversation as it ends. That grading reads transcripts, audio, tool calls and traces, across voice, chat, email and SMS, and until now every judged check came from a large language model reading that evidence and writing out its judgment.
Today we are launching Jev as a judge. Jev is TypeSafe’s decision model, and as of this week it is live on Bluejay as a grading option across our eval suite. It grades voice agents, chat agents and conversational agents on any channel we test, because it reads the same evidence our existing judges read: transcripts, tool calls and traces.
What is Jev?
Jev is not a language model in the usual sense. It cannot write a sentence. You send it a state, in our case the conversation, and a set of typed questions, and it returns typed answers: a yes or no with a probability, a choice from options you define, or a score on a rubric you describe. Every answer carries a probability distribution, and most carry a confidence value. That is the whole API, and it turns out to be most of what an eval suite needs.
Most grading questions are decisions
Look at what actually gets asked of a judge in Bluejay. Custom metrics are mostly pass/fail and yes/no questions. Tool call metrics check that the right call happened with the right arguments. Workflow adherence is a list of steps which the agent either followed or did not. Even scored metrics are bounded: frustration from one to five, resolution quality against a rubric you wrote. The possible answers are known before the conversation starts and the judge’s job is to pick one.
Where Jev is different is the shape of the answer. It returns the verdict itself, typed, with a probability behind it and nothing to parse, in well under a second, priced on input tokens only. The tradeoff is that it gives no rationale, and it only answers questions whose possible answers you defined ahead of time.
Some checks want exactly that shape but others might necessitate a written judgment, with the latter being a more traditional LLM use case.
Where it plugs in
Custom metrics already declare a response type. A metric that resolves to a bounded answer, pass/fail, yes/no, a choice from options you define, or a score inside set bounds, can use Jev on anything text-shaped: transcripts, tool calls, traces. Qualitative and JSON metrics, and metrics on the audio route, stay with the judges built for them.
Metric applicability has the same shape. Before grading, Bluejay checks whether a metric even applies to a conversation, which is itself a yes/no question, exactly the kind of high-volume bounded decision Jev is built for.
Confidence decides who judges
Every Jev answer tells you how concentrated its probabilities are. We use that signal instead of asking you to trust it blindly. Each Jev-graded metric has its own confidence threshold. At or above it, Jev’s verdict stands. Below it, Bluejay grades that one check again with an LLM judge.
One check on one conversation. Confident verdicts stand; everything else escalates to the LLM judge. Either way, one result is stored.
Two cautions. First, confidence is not correctness. A confident verdict can still be wrong, so the threshold saves cost but does not guarantee accuracy. Second, test Jev before you rely on it. Compare its verdicts with the judge you trust today, on conversations you have already graded. Then set each metric’s threshold where the two disagree. Metrics Lab in Bluejay is built for exactly that comparison.
Grade everything, not a sample
The reason to care is volume. Benchmarks put Jev at 20 to 200× faster and 40 to 400× cheaper than frontier models, and it is priced on input tokens only. Those are vendor figures, so treat them the way you would treat any benchmark that is not yours: benchmark the judges on your own graded conversations in Metrics Lab and look at the agreement. The order of magnitude is the point. We will also be publishing public judge benchmarks through Bluejay Labs at research.getbluejay.ai as we run them.
At that price, the habits that grew around costly grading stop making sense.
- Production monitoring grades every conversation. Sampling existed because grading every conversation used to be costly. Bounded checks on Jev are cheap enough to run on all of it, so an alert now fires on the conversation that failed, not on the sample that happened to include one like it.
- Verdicts land in seconds. A monitor that pages on a failed check pages moments after the conversation ends, because the verdict lands in well under a second.
- Load tests get graded for real. A thousand-conversation load test used to be graded on a slice. Now every conversation in the run carries its full set of bounded checks.
How Jev grades voice agents
Jev reads text, so for voice agents it grades the transcript, the tool calls and the trace, with the transcript coming from the same transcription and diarization output our other judges already use. Nothing about the voice pipeline changed. Latency, interruptions and talk-time stay measured from the recording, never judged, and audio-route metrics, the ones that listen to the audio file itself, stay with the audio judge.
The one thing to design around is state size. Jev’s window is smaller than a frontier model’s, and TypeSafe is open about accuracy dropping when the state is mostly irrelevant. A long support call is mostly irrelevant to any single check, so point each metric at the part of the conversation it is actually about. Shorter states also mean fewer input tokens, which is the whole bill.
How to turn on Jev in Bluejay
Jev grading is live in the product now, and there is nothing to connect: Bluejay manages the model access, so there is no API key to bring. Four steps:
- Open a custom metric, or create one, and choose Jev as its judge. Jev is available for pass/fail, yes/no, enum and quantitative metrics on the text route.
- Set the metric’s confidence threshold.
- Test the metric against recent conversations before committing, and compare its verdicts with the judge you use today.
- Run it. Experiments, scheduled simulations and production monitors pick up the new judge with no other changes, and results land in the same dashboards.
Conclusion
Most of what an eval suite asks a judge is a bounded question with a known set of answers, and that is the question Jev was built to answer: a typed verdict with a probability behind it, in well under a second, priced on input tokens. Jev is live in Bluejay today for voice, chat and conversational agents on every channel we test: pick it as the judge on a metric, set its confidence threshold, and Bluejay sends any check Jev is unsure about to an LLM judge. Your dashboards will not look different. The bill and the time-to-verdict will.
