Model upgrades are breaking changes

A provider ships a better model. Benchmarks improve across the board. You update the model string and your extraction accuracy drops on one document type that happens to be your largest customer's.

Nothing went wrong exactly. The new model is genuinely better on average. It is differently behaved on your distribution, and averages do not tell you about your distribution.

Why upgrades bite in unobvious ways

Prompts are tuned to a model, whether you meant that or not. Every instruction that was added because the old model got something wrong is now compensating for a failure that may no longer exist — and may be actively harmful. Prompt engineering is implicit fitting.

Output style shifts. Terser or more verbose, more or less inclined to hedge, differently structured. Anything downstream that parses loosely or matches on phrasing is exposed.

Refusal boundaries move. A newer model may decline requests its predecessor handled, or vice versa. If your domain sits near a safety boundary — medical, legal, financial, security — this can change a working feature into one that intermittently refuses.

Tool-calling behaviour changes. Which tools get chosen, how arguments are populated, how many calls per task. A change here alters cost and latency as well as correctness.

Cost changes in both directions. A cheaper per-token model that produces more reasoning tokens can cost more per task.

Treat it exactly like a dependency upgrade

Pin explicit versions. Never float. If your configuration names a moving alias, your production behaviour changes when the provider decides, not when you do. This is the single most important item on the list.

Run the evaluation suite against the candidate before switching. Per segment, not in aggregate — the whole point is that averages hide the regression you care about. A model that improves overall while dropping eleven points on one customer's document type has failed the test that matters.

Shadow it on live traffic. Send a share of real requests to both models, serve the old one, and compare. This catches distribution effects the suite misses, because production always contains inputs the suite does not.

Roll out gradually, with the old version reachable. A percentage of traffic, watched, with reversion as a config change rather than a deploy. The trace must record which model served each run or the comparison is unusable.

Re-test the prompt, not just the model. Instructions added to work around old failures may now be counterproductive. An upgrade is the natural moment to try removing accumulated compensation and see whether quality holds.

Deprecation is the other half

Providers retire versions. If your architecture assumes a specific model indefinitely, a deprecation notice becomes an unplanned migration on someone else's schedule.

The mitigation is the same interface discipline that makes provider switching possible: the model call behind your own boundary, the version in configuration, and an evaluation suite that can qualify a replacement in days rather than weeks. None of that is exotic. It is the difference between a scheduled task and a fire drill.

The question to answer now

If your provider deprecated your current model with ninety days' notice, what would you do, and how long would qualifying a replacement take?

If the answer involves a period of hoping, the evaluation suite is the thing to build first — not because upgrades are frequent, but because it is the only asset that makes them routine.