The cost model of an LLM feature, worked through

Most LLM cost surprises are not pricing surprises. The published rate was correct. What was wrong was the estimate of how many calls happen, how large they get, and how often the work is done more than once.

Here is the arithmetic to do before you build. All figures below are illustrative — substitute yours.

Start from runs, not tokens

The first number is not token price. It is runs per month, and it is usually underestimated because the estimate is made against the happy path.

A document pipeline processing 5,000 documents a month does not do 5,000 runs. It does:

• 5,000 extraction runs
• plus retries on failures — at a 6% failure rate, 300 more
• plus a validation or checking call if you have one, so double the above
• plus re-runs when a user corrects something and asks it to try again
• plus every run in staging, CI and the evaluation suite

That last line is the one that gets missed. An evaluation suite of 300 cases running on every merge, twenty merges a week, is 24,000 runs a month — potentially more than production. It is worth every penny and it must be in the model.

Then context, which grows

Input tokens are usually the larger half, and they inflate in ways nobody plans:

Retrieved context. Five chunks at 500 tokens is 2,500 tokens before the instruction. Raise k from 5 to 10 to improve recall and you have doubled the input cost of every call.

Conversation history. If a session accumulates, cost per turn rises through the session. Turn ten costs several times turn one. A long-running assistant has a cost curve, not a cost.

The prompt itself. A carefully developed system prompt with examples can reach a couple of thousand tokens, paid on every single call.

Tool definitions. Every tool schema is in the context. Fifteen tools with thorough descriptions is a fixed overhead on all traffic.

The multipliers people forget

Agent loops. A single user action becomes n model calls where n depends on how many tool calls are needed. Average of four, tail of twenty. Budget on the tail, not the average, because the tail is where the bill lives.

Reasoning output. Where a model produces internal reasoning tokens, they are billed and they are not visible in the answer. A short response can be an expensive call.

Reprocessing. Change the prompt, change the chunking, change the model — and any backfill re-runs the whole corpus.

What to do about it

Cache the stable prefix. If the provider supports prompt caching, structure the prompt so the fixed part comes first and is reused. On systems with a large system prompt this is frequently the single biggest reduction available, and it is a reordering rather than a redesign.

Route by difficulty. Most inputs are easy. Use a smaller model by default and escalate to a larger one on low confidence or on validation failure. The saving comes from the ratio, so this only pays if you measure what share of traffic actually needs the big model — and it is usually a small share.

Trim retrieval instead of raising it. Better chunking and hybrid search often let you lower k while improving recall. That reduces cost and improves quality at once, which is rare enough to be worth pursuing first.

Do not send what you will discard. Filter documents before embedding them in context. A cheap pre-filter is orders of magnitude cheaper than the tokens it removes.

The two guardrails

Whatever the model, two things must exist in code rather than in a spreadsheet: cost attributed per account or per feature, so you know where it goes, and a hard ceiling per run and per account, so a loop cannot become an unbounded bill.

Both are an afternoon of work at the start and considerably more once traffic is real.