How We Built Email Simulations

Customers reach a business on every channel it offers, and AI agents are now answering on all of them. Bluejay already tests those agents on voice, chat and SMS. Email is one of the busiest support channels there is, and more of each inbox is being answered by an AI agent every month. Those agents deserve the same testing as the ones on the phone.

Today, testing an email agent usually means posting text to the model through a webhook. That checks the writing. It misses everything between a customer's inbox and the agent, which is where most email problems happen. So we built email as a full channel in Bluejay. This post covers the architecture and the decisions behind it.

A few terms come up throughout. A Digital Human is a simulated customer with a name, a goal and a way of writing. A simulation runs a set of Digital Humans against one agent. Our running example is Lumen, an online furniture store whose support agent answers [email protected], and Maria, a Digital Human whose lamp never arrived.

Why a webhook test is not enough

Only one common email failure is about what the agent wrote.

What it had to do

• Real mail both ways: the Digital Human sends from a real mailbox and receives the agent's real reply.
• Many conversations at once: a simulation can run hundreds of threads in parallel, and every reply has to land in the right one.
• Either side can start: some agents answer customers, and some write first, like a dealership following up on a lead.
• Delivery facts: for every reply, we need the mail servers' own verdicts on authentication, spam and bounces.
• Email time: a turn can take minutes or longer, and nothing should hang when a reply never comes.
• Little setup: customers should only have to allowlist one domain.
• Production monitoring: the same loop has to run on a schedule and page someone when the channel breaks.

Architecture

Email simulations reuse most of what already powers voice and SMS. The Digital Human engine, the grading pipeline and the uptime monitors are shared. What we added is a mail layer with an outbound path and an inbound path.

• Digital Human: the same engine that runs SMS conversations writes each email. It never needed to know the channel changed.
• Send queue and worker: every outgoing email becomes a job. A separate worker sends jobs at a steady rate and retries failures.
• Mail service: sends and receives on our own domain and reports what happened to each message, including bounces and complaints.
• Inbound mail: stores every reply exactly as it arrived, then sends us a signed notification.
• Router: verifies the notification, decides whether the reply counts, and finds the conversation it belongs to.
• Reader: pulls the agent's words out of the message and hands them to the Digital Human for its next turn.
• Grading and results: delivery checks run as each reply arrives. The content grade runs once at the end. Everything lands in the same results, dashboards and alerts as voice and SMS.

Key decisions

Run our own mail stack

Our first version sent through an email API from one shared sender address. It worked for a single test and broke down at scale in two ways. The provider's rate limit applied to the whole account, so parallel conversations throttled each other. And one shared address pooled every customer's sending reputation into one place we could not control.

We compared the options against what the product needed.

One permanent address per Digital Human

All test mail comes from mail.bluejaysims.com, a subdomain we keep for testing. We never send tests from getbluejay.ai or from a customer's domain, so a test can never hurt anyone's real reputation. Customers allowlist it once.

Each Digital Human gets a permanent address on that domain, just like the phone number it uses on voice and SMS tests. The number in the address identifies the Digital Human and never changes.

1[email protected]

Our first design gave every conversation a fresh address. That broke outbound tests, because a customer's CRM needs an address it can keep on a lead between runs. A permanent address fixes that, and the same address works whichever side writes first.

Routing stays simple because a Digital Human holds one open email conversation at a time. A reply to Maria's address can only belong to Maria's open thread, even if a help desk tool rewrites the subject line. To run more threads in parallel, a simulation adds more Digital Humans. We also set the standard reply headers on every email so the thread looks normal in the agent's mailbox.

Send through a queue

The app never sends mail directly. It writes a job for each turn and moves on. A separate worker sends the jobs.

• Pacing: the worker sends at a steady rate under a shared limit, so a large run never goes out as a burst. Bursts are what spam filters look for.
• Exactly once: each job is keyed by conversation and turn. A retry or a duplicate job can never send Maria's email twice.
• Isolation: a slow mail provider never slows the app. Failed jobs retry with backoff, and jobs that keep failing are set aside for review.

Keep the raw message

Every reply is stored exactly as it arrived. That one choice drives most of the inbound design. The mail servers' verdicts are in the headers, so the delivery checks read facts instead of guessing.

1From: Lumen Support <[email protected]>2To: Maria Chen <[email protected]>3Subject: Re: My lamp never arrived4Authentication-Results: mx.mail.bluejaysims.com;5 spf=pass smtp.mailfrom=lumen.example;6 dkim=pass [email protected];7 dmarc=pass header.from=lumen.example8X-Spam-Status: No

The raw message also lets us read replies properly. Most arrive as a multipart message, a bundle of versions and attachments wrapped together.

1multipart/mixed2├─ multipart/alternative3│  ├─ text/plain          read this first4│  └─ text/html           used when there is no plain version5└─ application/pdf        receipt.pdf, skipped

We walk that structure and take the plain text version when there is one. When a reply only has HTML, we turn it into text and keep its paragraphs and sign off. Then we cut the quoted thread under the new text, so the agent never gets credit for words Maria wrote. Every stored message is keyed by its unique message ID, so a notification delivered twice never becomes two turns.

Trust nothing by default

The inbound path is open to the internet. Two checks protect it. First, every notification is signed, and we verify the signature against the mail service's own certificate before reading anything. Second, every reply has to pass one rule before it can touch a conversation, because anyone can type any From line.

Design for email time

A voice turn takes a second. An email turn can take an hour if a person approves replies. So each conversation has a quiet timer that resets on every reply. When it runs out, the conversation ends and gets graded.

A timer alone was not enough. If the first email never went out, no reply would ever arrive to start the timer, and the run would wait forever. So every conversation also gets a hard deadline when it starts, and a background sweep closes anything past it.

Grade each check the way it should be graded

• Delivery by rules: SPF, DKIM, DMARC, spam and bounce are facts the servers already decided. We record them as each reply arrives. A model would only add a chance of getting a fact wrong.
• Latency recorded, never graded: a good reply time differs by team, so we record it and let each team set its own bar.
• Content by a model, on the whole thread: an email answer only makes sense next to what came before, so the content grade runs once the conversation ends.

Protect the domain

If our domain started landing in spam, every customer's tests would start failing because of us. Keeping it healthy is what keeps a failing test pointed at the agent.

What we learned in production

• Quarantine is the most common failure: corporate mail systems hold unfamiliar senders before they reach the mailbox. When a test email does not arrive, it is almost always sitting in an admin quarantine.
• Staging domains rarely have DMARC: one team's pre-production mail server had no policy at all. Checking DKIM alignment ourselves let their tests run without a DNS change.
• Agents reply from more than one address: help desk tools often answer from a different address than the one customers write to. That is why the trust rule flags these replies instead of dropping them.
• Quoted text comes in many shapes: some mail clients wrap the "On Monday, Maria wrote" line across two lines. Our first version missed it and graded quoted text as the agent's words.

Conclusion

Most email agent failures happen in delivery and authentication. We built email simulations to test that whole path. One shared domain keeps trust in one place, a paced queue keeps it healthy, raw messages give us the servers' own verdicts, and a strict trust rule makes sure every turn came from the agent. The Digital Human engine, grading and monitoring are the same ones that run voice and SMS, so a team can now test one agent on every channel its customers use, from the same place.

To try it, create an agent, choose Email, and enter the inbox your agent answers from. The email agent docs cover setup and the API.