Why your evaluation suite passes and production still breaks

A team ships an agent. The evaluation suite is green — ninety-something percent on a set of a few hundred cases, held steady across three model versions. Two weeks after launch, support is fielding complaints about outputs nobody has seen before.

The instinct is to blame the model. It is almost never the model. The suite passed because it measured the thing the team already understood, and production is made of the thing they did not.

An eval set is a sample, and you chose it

Evaluation sets get built one of three ways. Someone writes cases by hand. Someone exports a slice of historical data. Someone generates cases with a model. All three share a property that matters more than any of their differences: the person selecting the cases is the person who built the system.

That is selection bias with a very short feedback loop. You write cases for the behaviours you thought about. You fix the failures. The suite goes green. What you have measured is the intersection of the system's competence and your imagination — and you have no reading at all on the part of the input space that never occurred to you.

Production does not sample that way. Production samples from whatever your users actually do, which includes the things they do wrong, the things they do at 2am, and the things they do because a form on some other system let them.

Four specific gaps

1. The input distribution is different, and not randomly

Hand-written cases are tidy. They have the fields filled in. The dates parse. The PDF has a text layer. Real inputs arrive with a scanned page rotated ninety degrees, a currency symbol in a quantity column, and a note in the description field that contradicts the structured data next to it.

This is not an edge case in the statistical sense — it is not rare. It is a systematically different distribution, and a suite drawn from clean data tells you nothing about it.

2. Nobody writes the degenerate cases

The empty input. The input that is one character. The input that is four megabytes of pasted log output. The input in a language the system was never scoped for. The input that is the previous output, because a user copied it back in.

These are boring to write and they are where the ugly failures live: unhandled exceptions, silent truncation, a retry loop that never terminates, a cost spike.

3. Upstream changes and nothing tells you

Your evaluation set is frozen. The systems feeding it are not. A field that was always populated starts arriving null a quarter of the time because someone made it optional in a form you do not own. An API adds a field, and your parser — which was tolerant, so it did not error — starts putting the wrong value in the right slot.

A frozen eval set cannot see this by construction. It will keep passing, accurately, on data that no longer resembles what is arriving.

4. The long tail is the product

For a lot of this work, the common case is the part nobody needed software for. If ninety percent of invoices are simple enough that a template would have handled them, then the value of the system is entirely in the remaining ten percent — and an aggregate score weights that ten percent at ten percent.

You can hold a suite at 94% while getting monotonically worse at the only cases that justified the project.

What to do about it

None of this argues against evaluation. It argues against believing a single aggregate number.

Sample from production, continuously. The eval set should be fed by real traffic, not only by cases someone wrote. Take a sample every week, label it, and fold it in. The suite should be growing in the direction production is moving.

Hold out by time, not at random. A random split leaks the future into the training and tuning of your prompts and thresholds. Splitting by time gives you an honest read on whether last month's tuning survives this month's inputs.

Keep every real failure forever. When something breaks in production, the case goes into a permanent regression set before the fix goes in. This is the single highest-value set you will own, because every case in it is a thing that actually happened rather than a thing someone imagined.

Score by segment, and look at the worst one. Aggregate accuracy hides exactly what you need to see. Break the set down by whatever dimension carries risk — document type, customer size, language, source system — and treat the worst segment as the number that matters. A system that is 97% overall and 61% on one customer's documents has a problem that "97%" will never surface.

Measure the refusals. A system that escalates to a human when it is unsure is behaving correctly, but an escalation rate that climbs from 4% to 19% is a signal about drift that accuracy alone will not give you. Track it as a first-class metric.

The uncomfortable framing

An evaluation suite is not a measurement of how good your system is. It is a written record of what you have thought of so far.

Treated that way, the number stops being reassuring and starts being useful: the question is no longer "are we at 95%" but "what did we add to the suite this month, and where did it come from". If the answer is "nothing, and nowhere", the suite has stopped doing its job — regardless of what it says.