How We Improved LLM‑as‑a‑Judge Accuracy with Atomic Success Criteria

Every simulation on Bluejay ends in a verdict. A Digital Human runs the conversation, and an LLM-as-a-judge reads the transcript, the tool calls, the trace and, for voice agents, the audio recording itself, then grades the agent against the test case’s success criteria. Until recently those criteria were one field: a single compound criterion, a paragraph stating everything a passing conversation looks like, matched by default on every test case.

We changed the shape of that field. Success criteria are now atomic: a list of short, targeted checks, each naming exactly one behavior you expect from the agent, each graded independently by the judge. On paper it is a small change, one text box becoming several. In practice it moved the number we care about most: how often human reviewers agree with the judge.

Why a compound success criterion is an underspecified eval

Here is the shape the old field encouraged, for an agent that refills prescriptions:

The agent should verify the caller’s date of birth, locate the active
prescription, offer store pickup and delivery, place the refill order,
and send a confirmation text before ending the call.

Grammatically that is one criterion. Logically it is a conjunction of five predicates, and the question the judge is asked, did the conversation meet this criterion, never says how to aggregate them. Suppose the agent verified the date of birth, found the prescription and placed the order, but never mentioned delivery. The truth value of each predicate is perfectly clear, and the verdict still is not, because the aggregation function is unspecified. Treat the conjunction as strict and the conversation fails. Weight the order above the pickup options and it passes. Both readings are defensible, so the judge fills the gap with one prior, the human reviewing the result fills it with another, and the disagreement gets filed as a judge error when it is really a specification error.

The failure scales with thoroughness. Every clause folded into the compound adds a predicate whose weight is undefined, so the most carefully written test cases, the ones whose criteria tried to cover everything, were exactly the ones where the judge and our customers’ reviewers parted ways most often.

What are atomic success criteria?

Atomic success criteria are short, independently graded checks, each testing exactly one observable behavior of the agent: one predicate per criterion, one verdict per predicate. Here is the same test case, decomposed:

- Verifies the caller’s date of birth before discussing the account
- Locates the caller’s active prescription
- Offers both store pickup and delivery
- Places the refill order
- Sends a confirmation text before the call ends

Each line is one atomic criterion: a single predicate over the conversation, graded on its own. Instead of one broad judgment call, the judge runs five narrow binary checks, each answerable from the transcript, the tool calls, the trace or the audio, and returns a verdict per criterion. No aggregation happens inside the judge anymore; the roll-up happens outside it, in the open, where you can see exactly which check failed. The change applies anywhere success criteria live on the platform, on test cases you wrote, on test cases we generated, and on the Digital Humans that run them.

Why do atomic criteria make an LLM judge more accurate?

An LLM judge errs where the question leaves it degrees of freedom. A compound criterion leaves several: which predicates carry the most weight, what to do with a conversation that satisfies some clauses and not others, how much generosity “should” buys. An atomic criterion leaves almost none. “Offers both store pickup and delivery” is a narrow classification over evidence the judge can point to, with barely any interpretation left over. You fixed the aggregation semantics when you wrote the criteria; the judge only decides whether each named behavior happened.

Isolation matters as much as size. Under a compound criterion, one misjudged clause corrupts the only verdict there is. Under atomic criteria, an error stays local to its check: four correct verdicts survive a fifth wrong one, and partial credit is visible instead of collapsed. A conversation that misses one criterion of five is a different situation from one that misses four, and the old shape graded both the same FAIL.

How we measure LLM-as-a-judge accuracy

Accuracy for an LLM judge has no ground truth except people, so we measure it the way the field does: agreement between the judge’s verdicts and human review of the same conversations. We compared that agreement on either side of the switch, with the judges, the models and the evidence they read unchanged; the only variable was the shape of the criteria. Agreement went up. The more granular the criteria, the more often human reviewers signed off on the verdicts the AI evaluators produced.

That is the property an automated eval lives or dies by. A judge the team second-guesses gets re-checked by hand, and grading that gets re-checked by hand is not automation, it is overhead with extra steps. The way to earn that trust turned out not to be a smarter judge but a better-specified question.

Easier to read, easier to write

The agreement number was the point of the change. The day-to-day win is legibility. Open a test case and the criteria tell you at a glance what it is evaluated on; the same goes for a Digital Human. When a test fails, the failure names the criterion. Delineating that specific failure makes it easier to identify exactly what went wrong than to navigate through a block of text and through the email, so you can fix a larger failure or a specific one. You start from “the agent never offered delivery.” And when an expectation changes, you edit one line instead of re-drafting the sentence that four other expectations live in.

You can see direct improvement on specific success criteria now. That is what lets you go in and look across a multi-dimensional criterion: each check is its own dimension, so a change that moves one of them shows up on that check.

How to write atomic success criteria

Atomic criteria reward a different writing habit than compound ones. What we have converged on:

• One predicate per criterion. If a line contains an “and”, it is usually two criteria.
• Keep it observable. A criterion should be decidable from the transcript, the tool calls, the trace or the audio recording. “Places the refill order” is a tool call; “speaks the confirmation number clearly” lives in the audio; “understands the caller” is not evidence of anything.
• Name the behavior, not the vibe. “Confirms the date and time back to the caller” beats “is helpful and thorough”, for every judge, human or not.
• Shorter is better. Specificity carries the information, not length. A criterion that runs past a line is probably hiding a second one.

Conclusion

Decomposing one field into atomic criteria looks like a UI tweak. It is really a change in the contract with the judge: grading moved from one broad question with implicit aggregation to a set of narrow, independent checks over the behaviors you actually care about, each with less room for error and a verdict a human can audit at a glance. Agreement between human reviewers and the AI evaluators went up, and the tests got easier to read and write along the way. A failed check names the behavior, so you can see exactly what went wrong, and you can see that specific criterion improve. Atomic success criteria are live on Bluejay today, for test cases and Digital Humans, across every channel we test.