Evaluating agents in production

Most teams building agents can tell you their system works. Very few can tell you whether last week's change made it better. The difference between those two positions is an evaluation practice, and it is the single largest predictor of whether a system survives contact with real users.

This paper describes the method we use. It assumes no particular framework and no particular model provider. It assumes only that you have a system that takes input, does something, and produces output that a person could grade.

What evaluation is for

The purpose of an evaluation suite is not to produce a score. It is to answer one question, repeatedly and cheaply:

Is this version better or worse than the one currently running?

Everything else follows from taking that question literally. It implies a fixed comparison basis, so the sets cannot be regenerated casually. It implies a decision rule, so somebody has decided in advance what "worse" means. And it implies the suite runs before a release rather than after an incident.

A score with no decision attached is a vanity metric. If 91% and 88% would both result in shipping, you did not need the number.

Building the set

Four sources, in order of value

Production failures. The highest-value cases you will ever own, because each one is a thing that actually happened. Every incident, complaint and manual correction becomes a permanent case, added before the fix ships so the case genuinely fails first.

Production samples. A weekly random sample of real traffic, labelled by someone who knows the domain. This is what keeps the suite tracking reality rather than tracking the team's assumptions.

Hand-written adversarial cases. Empty inputs, enormous inputs, wrong-language inputs, inputs containing text that looks like instructions, inputs that are a previous output fed back in. Boring to write, disproportionately where the ugly failures live.

Generated cases. Useful for volume and for systematically varying one dimension. Weakest source, because generated cases inherit the generator's blind spots, which correlate with your own.

Three sets, kept apart

Development — you look at it constantly while iterating. Assume it is contaminated by your own tuning and treat its numbers as directional.

Regression — every historical failure, permanent, append-only. This set only grows. A case is never removed because it has become inconvenient.

Holdout — sampled from a later time window than anything used for tuning, looked at rarely, and never used to make a prompt decision. This is the set that tells you the truth.

The time-based split matters more than people expect. A random split leaks the future into your tuning: you end up choosing prompts and thresholds partly on cases drawn from the same week as your test data. Splitting by time answers the question you actually care about — does tuning done in March survive the traffic of April.

How big?

Big enough that the difference you care about is larger than the noise. If you need to detect a five-point change and your grading is itself noisy, a set of thirty cases will tell you nothing you can act on. A few hundred well-chosen cases per segment beats several thousand generated ones, and is cheaper to maintain.

What to measure

Task success, defined before you measure it

The definition is the hard part and it is domain work, not engineering work. "Extracted the invoice correctly" needs to resolve into something specific: which fields are mandatory, whether a formatting difference counts, whether a confident wrong answer and an explicit refusal score the same. Write it down. Two people grading the same case should reach the same verdict.

Grading methods, and their costs

Exact or structural match where the output is structured. Cheap, deterministic, and the right default whenever it applies.

Programmatic assertions for properties rather than values: the total equals the sum of line items, no field references a document that was not supplied, the output parses.

Model grading for open-ended output. Workable, but it is a system with its own error rate and its own drift. If you use it, calibrate it against human labels periodically and report that calibration alongside the score. An uncalibrated model grader is a number generator.

Human grading as the anchor. Expensive, so use it on a small rotating sample rather than the whole set — its job is to keep the cheap methods honest.

The metrics that earn their place

• Success rate by segment. Never report only the aggregate. Break it down by document type, customer, language, source system — whatever carries risk — and treat the worst segment as the headline.
• Escalation rate. How often the system correctly declines. A rise is a drift signal long before accuracy moves.
• Confidently wrong rate. Wrong output with no escalation. This is the number that damages trust, and it should be tracked separately from ordinary failure.
• Cost and latency per run at median and 99th percentile. A change that improves accuracy by a point and triples cost is a decision, not an improvement.

Gating a release

The suite runs in CI on every change that can affect behaviour — and that includes prompt edits, model version changes, tool schema changes, and retrieval configuration, not only application code.

A workable default policy:

• Regression set: zero new failures. A previously fixed bug reappearing blocks the merge, without discussion.
• Holdout: no segment may drop by more than an agreed margin.
• Confidently-wrong rate: must not increase.
• Cost per run: increases above an agreed threshold require an explicit sign-off rather than a silent pass.

Publish the per-segment table on every run, not just the pass/fail. The table is what makes a bad trade visible; the boolean hides it.

Keeping it honest over time

Evaluation suites decay in three predictable ways.

They get tuned against. After enough iterations, prompts are shaped to the development set and its score stops carrying information. Countermeasure: the holdout, looked at rarely and never used for tuning decisions.

They go stale. Production moves; the frozen set does not. Countermeasure: continuous sampling, and a standing expectation that the suite grows every month. If nothing was added, that is a finding.

They get quietly weakened. A case that keeps failing gets marked flaky and skipped. Countermeasure: removals require the same review as a code change, and the regression set is append-only by policy.

What this costs

Honestly: the first version is one to two weeks of work, most of it spent defining success rather than writing code, and it needs a domain expert's time rather than only an engineer's. Ongoing, it is a few hours a week of labelling and review.

That is a real cost and it is worth stating plainly, because the alternative has a cost too — it is just deferred, unbudgeted, and paid during an incident.

Where to start

If you have nothing today, the smallest useful version is a day's work:

1. Collect fifty real inputs, including ten that are known to be awkward.
2. Write down what a correct output is, precisely enough that two people agree.
3. Grade the current system by hand and record the number.
4. Put the cases in version control next to the code.
5. Run it before the next change ships.

That is not a mature practice, but it converts "it seems fine" into a number with a date on it — and everything else in this paper is an extension of that single move.