Benchmarking LLMs in a User Simulator

Bluejay tests agents by calling them with simulated people. We call these Digital Humans.

A Digital Human will hold a conversation with an agent the same way a real person would.

When it comes to running a test (simulation), the test is only good when both the agent and the Digital Human work. Thus if the Digital Human messes up, the test evaluates to still fail the agent even though it is not the agent’s fault!

Our goal is to decide which LLM simulates a Digital Human best on quality, latency and cost.

GPT 6 Sol scored highest at 82.0%, while the Gemmas replied fastest but scored lowest.

This post walks through how we built the benchmark, the tradeoffs we made, and what we found.

Digital Human Configuration

Each Digital Human field has a pass or fail check worth points, and a conversation’s score is the points it earned out of the points possible.

Enum-based fields, which include but are not limited to:

Free text instructions, which include but are not limited to:

Building the benchmark set

Anonymization

We built the set from anonymized Digital Human configurations across many industries, creating an initial pool of over 5,000 Digital Humans. To remove real details without adding bias:

• We label each detail by kind, e.g. a person, a company, an address.
• We replace each detail with one ordinary, gender neutral value, e.g. every name becomes Jordan Taylor and every phone number 123-456-7891.
• A second LLM judge fails the set if any real detail is left.

Creating a diverse set

We wanted 500 diverse Digital Humans that represent the tests we’ve seen. To pick them, we first needed a number for how different two Digital Humans are.

The difference score

We scored every possible pair in the pool from 0 to 1:

where

• $f$ is each free text field both Digital Humans set
• $p_f$ is the field’s weight, e.g. 0.3 for the intent and 0.15 for the role. These weights are only for picking the set, not for scoring.
• ${\mathrm{sim}}_f$ is how similar the two texts are

Text similarity

Embedding cosine is the default way to compare text, so we tested it first on four intents (shortened here):

1. Make a dental claim. If the agent can’t get your name right after you spell it, just say yes.
2. Make a medical claim, with the same extra line.
3. Make a claim.
4. Make a dental claim.

Embedding cosine scored every pair between 0.96 and 0.99. It can’t tell a one-word change that changes the test from a whole extra instruction.

So we use each method where it works:

• Phrase overlap spreads the pairs out, so we use it to remove duplicates.
• Embedding cosine stays in the difference score, after we subtract the average embedding so unrelated texts stop looking alike.

Choosing the set

Now we had over 5,000 Digital Humans and room for 500.

To measure a set, we took how close every Digital Human in the pool sits to its nearest pick, and compared that to a random sample of the same size. We explored a few paths:

• Stratified sampling from each enum field (e.g. language). No better than random, missing rare values.
• Shrinking each Digital Human to a few numbers first (UMAP, spectral embedding). The more it shrank, the worse the set.
• Picking the most different Digital Humans. This chases outliers such that the average Digital Human ends up farther than random samples.
• Clustering, then picking a representative Digital Human per cluster. Better than random!

Each method below keeps one central Digital Human per group.

We ended up using greedy facility location, scoring similar to FasterPAM. It starts with an empty set and adds one Digital Human at a time, each time picking the one that brings the rest of the pool closest to the set.

Adding one at a time let us skip any Digital Human whose source already filled 10% of the set. FasterPAM can’t do this, because it starts with a full set and keeps trading one pick for another, so there’s no point where it adds a Digital Human and can check the rule.

Two ways to score

We ran the 500 Digital Humans on each model and scored the conversations two ways.

Relative scoring

With this scoring method we run a blind pairwise comparison.

1. Each Digital Human runs once on our baseline (GPT 5.4) and once on the candidate.
2. The transcripts are anonymized (model removed).
3. Two frontier LLM judges from different vendors read both transcripts with the Digital Human’s fields and pick which model followed each instruction better.
4. Each judge grades each pair twice, with the order swapped.
5. If the two grades disagree, it’s a tie. The score is wins plus half the ties, so 50% means no difference.

The advantage of this scoring method is that judges always have a reference point. This lets the judge catch smaller differences than an absolute scorer.

Absolute scoring

With this scoring method we grade each conversation on its own against a fixed checklist.

1. Each Digital Human runs on each model.
2. The transcripts are anonymized (model removed).
3. Two frontier LLM judges from different vendors grade every check in the configuration tables as pass or fail.
4. A check whose moment never came, like a hangup rule in a call that never got there, isn’t counted in the score.
5. The score is points earned out of points possible, averaged over the two judges, with a 95% range from resampling the Digital Humans.

The advantage of this scoring method is that every model lands on one scale to hill climb. We can compare every model at once and see how far each one is from 100%.

In our benchmarking, we use relative scoring to accurately decide if a candidate beats an incumbent model, and an absolute score to set a north star.

Results

Notes:

• Reply times are per Digital Human reply, pooled over every conversation.
• Qwen3.8-27B and both Gemmas run on our own B200 GPUs with 8-bit (FP8) weights, so they have no per-call price.
• Qwen’s average sits above its p95 because a few of its replies took over 18 seconds.
• In relative scoring against GPT 5.4, GPT 6 Sol did best at 56% and Gemma 4 26B worst at 39% to 44%.

Takeaways

1. Benchmark your methods to a simple baseline continuously for best results.
2. Use relative scoring to pick a winner and absolute scoring to see how far every model is from good.
3. Following the intent is the most difficult area for Digital Humans.