Observability for LLM systems: what to log when the output is text

Standard application monitoring answers: did it respond, how fast, did it error. For a language model feature, all three can be green while the system is producing nonsense.

The failure that matters returns HTTP 200, in 900 milliseconds, with a confident and incorrect answer. No conventional alert fires, because nothing conventional went wrong.

What a trace has to contain

Six weeks after a run, someone will ask why the system did a particular thing. To answer, a trace needs:

• The inputs as received — the actual document, the actual query, not a summary
• Retrieved context, with identifiers — which chunks, from which documents, at which versions
• The prompt version — a hash is enough, but without it the trace records what happened and not why
• Model and parameters — including the exact version string, not "the latest one"
• Every tool call with resolved arguments — the values actually used, not the intent
• The raw output before parsing — because parse and validation failures are invisible once discarded
• The decision — accepted, retried, escalated, refused, and why
• Cost and token counts — per call, so aggregation is possible later

Two of these get skipped most often and cost the most. Prompt version, and raw output. Without the first you cannot correlate a behaviour change to a change. Without the second you cannot tell a model that produced malformed output from a parser that mishandled valid output.

The metrics that actually move before an incident

Latency and error rate are necessary and insufficient. Add these:

Validation failure rate, per field. A field that starts failing more often is the earliest signal you get that inputs have shifted — usually before quality is visibly worse.

Escalation rate. How often the system hands off. A rise means the system is encountering more it cannot handle, which is drift showing up as caution rather than as error.

Refusal rate. How often a tool declined a request. Rising refusals mean the model is increasingly attempting things it is not permitted to do — worth knowing early.

Confidently-wrong rate. Sampled and human-labelled, because nothing else can measure it. This is the number that damages trust and the only way to get it is to look at real outputs on a schedule.

Cost per run, at the 99th percentile. The median is stable and uninformative. The tail is where loops show up.

Sampling, and the retention problem

Full traces of every run are expensive and contain customer data. Two rules make this tractable.

Sample the successes, keep all the failures. A few percent of successful runs is enough to detect drift statistically. Every failure, escalation and validation error is kept in full, because those are the ones you will investigate.

Then set retention per artefact, not per system. Prompts, inputs, outputs, and derived metrics have different sensitivities and different useful lifespans. Aggregated metrics can live for years; raw inputs containing personal data usually should not. Deciding this once, explicitly, is much easier than reverse-engineering it during a security review.

The check that tells you if this is real

Pick a run from a month ago. Try to answer, from your logs alone: what did it see, what did it decide, which prompt and model version produced that, and would the same input produce the same result today?

Most teams discover they can answer two of the four. The gap is the work.