Structured output is the difference between a demo and a feature

The gap between a promising prototype and something you can ship is often exactly this: the prototype returns text that a human reads, and the feature needs an object that code can act on.

It sounds like a formatting detail. It is the point at which a language model stops being a demo and starts being a component with a contract.

Why "return JSON" is not enough

Asking a model for JSON in the prompt gets you JSON most of the time. Most of the time is the problem. The failures are specific and each needs handling.

Valid JSON, wrong shape. A field is a string where a number was expected, an enum contains a value not in the enum, a nested object is flattened. It parses and then breaks three functions downstream.

Hallucinated fields. The model helpfully adds confidence or notes. Harmless until something iterates over keys.

Plausible nulls. The field is missing from the source document, so the model invents something reasonable rather than returning null. This is the dangerous one, because the output is well-formed and wrong, and nothing downstream can tell.

Silent truncation. A long document produces twelve line items where there were nineteen. The JSON is valid. The array is short. Nobody notices until a total does not match.

The contract, in layers

Use the provider's constrained decoding. Where a schema-enforced mode exists, use it. It eliminates the parse-failure class entirely, which is not most of your problem but is the easiest part of it to delete.

Validate against a schema you own. Provider enforcement covers structure. It does not cover whether the values make sense. Parse the response into your own schema — the same one your database and API use — and treat a validation failure as a normal, counted outcome rather than an exception.

Make "not present" expressible. Every optional field needs an explicit way to say the source did not contain it. If the only options are a value or a missing key, the model will produce a value. A nullable field with an instruction to use null when absent removes an entire category of invention.

Assert invariants in code. The things that must be true regardless of what the model said: the line items sum to the total, the end date is after the start date, every referenced document was actually supplied. These are cheap, deterministic, and they catch the failures schemas cannot express.

Count, do not just retry. A validation failure that triggers a silent retry hides the signal. Record how often each field fails validation. A field that fails 8% of the time is telling you the source data is more varied than the schema assumes — which is information about your domain, not about the model.

The extraction case, specifically

For document extraction, add one more layer: make the model say where it found each value. A page number, a line, a quoted span.

This costs tokens and buys two things that are hard to get any other way. A reviewer can check a value in seconds instead of rereading the document. And a claimed source that does not exist in the input is a detectable fabrication — you can verify the span appears in the text you sent.

What this changes about the build

Once the model returns a validated object, the AI part of the system stops being special. It is a function with a typed return, a measurable failure rate, and a defined behaviour when it fails. It can be tested, queued, retried and monitored with the same tools as everything else in the codebase.

That is the actual goal. Not a cleverer prompt — a boundary boring enough that the rest of the system does not have to know a model is behind it.