Everyone agrees agents need evals. Far fewer people say where the evals come from when the workflow you're automating has never been written down.
And it usually hasn't. The process lives in the head of someone who's done it for nine years, who will describe it in four steps, and whose four steps omit every judgment that actually makes them good at it. Ask them to document it and you get a flowchart that's true on Tuesdays.
This is the method we use. It takes about a week of calendar time and it produces something you can regression-test against.
Step 1: Take the artefacts, not the description
Do not start with an interview. Start with the last hundred times the work happened.
The sent-messages folder. The closed tickets. The signed-off assessments. Whatever the workflow physically produces, in the form it actually took, including the sloppy ones.
This is the single most important decision in the process, and here's why: the description is what people believe they do; the artefacts are what they did. The gap between those two is exactly the space where an agent fails, so an eval set built from the description will systematically miss the things that will break in production.
Take a hundred. Not ten — ten is anecdote, and it hides the rare-but-recurring cases that turn out to be twelve percent of volume. If a hundred aren't available, take everything there is and note the limitation honestly.
Step 2: Cluster before you label
Now sort them, and resist the urge to sort by outcome. Sort by shape of input.
You'll typically find five to nine clusters, and they are not the ones on the org chart. Something like: the straightforward request; the request missing a critical detail; the multi-part request pretending to be one question; the returning customer with history that changes the answer; the one that isn't actually this workflow and was mis-filed; the genuinely novel case.
Cluster sizes matter enormously. That last cluster is often two out of a hundred, which tells you it should escalate rather than be handled — a decision you'd never reach by reasoning about it in the abstract.
The mis-filed cluster is usually the biggest surprise. Nobody mentions it in the interview because handling it is invisible reflex, but it's frequently ten to fifteen percent of what arrives, and an agent that doesn't know to reject-and-reroute will confidently do the wrong workflow.
Step 3: Interview against specific cases
Now talk to the expert — with printouts.
Not "how do you handle enquiries." Instead: "here's case 34. You replied like this. Why this and not the obvious other thing?"
The difference is total. General questions get you the sanitised process. Specific artefacts get you the actual rule: "Oh — that's because they're on the old contract. You can tell from the reference number prefix. Anyone with that prefix has different notice periods, we've never updated the system for it."
That sentence is worth more than a day of process mapping, and it will never come out of a general question, because the expert doesn't know it's unusual. It's just Tuesday to them.
Budget two or three sessions of an hour. Record them. The tacit rules come out sideways, in asides, usually while they're apologising for the mess.
Step 4: Write the cases as input plus assertions
For each case, capture the input exactly as it arrived — typos, missing punctuation, the screenshot instead of text — and then, instead of a single "correct output," write assertions.
This matters because for most real workflows there is no one right answer. There are many acceptable replies and a smaller set of unacceptable ones. An eval that demands an exact string will fail on outputs that are perfectly good, and you'll learn to ignore it, which is worse than not having it.
So: must cite the current fee schedule. Must not commit to a date. Must ask for the unit number, because it's missing. Must escalate rather than answer. Must stay under 80 words. Must name the specific product, not the category.
Assertions are checkable — some by string or schema rules, some by a model grading against a rubric, some by a human on the small subset where it genuinely needs a person. Mix freely. A cheap deterministic check on the top three assertions catches most regressions.
Step 5: Split by what the case is for
Not every case plays the same role, and mixing them produces a number that means nothing.
- The core set — the biggest clusters, the everyday work. These must pass, always. A regression here blocks release.
- The hard set — known-difficult cases you're consciously working on. Track the trend, don't gate on it.
- The escalation set — cases where the correct behaviour is not answering. The most-skipped category and among the most valuable, because an agent that never escalates is not safe, however good its answers are.
- The poison set — prompt injection attempts, off-topic requests, anything adversarial. Small, permanent, non-negotiable.
One aggregate accuracy number across all four is a number you can game and can't act on. Four scores tell you what to do next.
Step 6: Let it keep growing
The eval set built in week one is the seed, and it will be wrong in ways you can't yet see.
The mechanism that fixes that: every escalation and every rejected draft in production becomes a candidate case. When a human overrides the agent, that's a labelled example of the agent being wrong, generated for free, with the correct answer attached by the person who overrode it.
Which is the strongest argument for running at level 2 — draft and hold — for longer than feels necessary. It isn't just a safety posture. It's a data collection strategy, and it's the only source of eval cases that reflects what's actually arriving now rather than what arrived last quarter.
What you have at the end
A week in, you have somewhere between eighty and two hundred cases, sorted into four sets, each with assertions, running automatically on every change.
That artefact is what makes everything afterwards tractable. Prompt changes get a number. Model swaps become an experiment rather than a leap of faith. The autonomy conversation gets evidence. And the expert whose nine years of judgment lived in their head now has some of it written down — which, for most of our clients, turns out to be worth having regardless of what happens with the agent.
Friday: the approval queue is the interface.