Where an LLM belongs in a workflow, and where it does not

Most disappointing AI features are not badly built. They are correctly built in the wrong place.

A language model is a component with a specific shape: extremely good at interpretation and generation, unreliable at arithmetic and lookup, non-deterministic by construction, and priced per call. Put it where those properties are assets and it looks like magic. Put it where they are liabilities and no amount of prompt work saves it.

The one question worth asking

For any step in a process: what happens when this is wrong, and who finds out?

That single question sorts most candidate use cases without any experimentation at all.

If a wrong answer is caught immediately by the next step, by a schema, by a person who was going to look anyway — the model is a good fit, because its errors are cheap and visible. If a wrong answer flows silently into a system of record, an invoice, or a customer's inbox, the model is a poor fit for that position, however well it performs in testing.

Four positions that work

Interpretation at the edge. Turning something unstructured into something structured: a forwarded email into a ticket with fields, a scanned page into line items, a rambling request into a category. The output is immediately checkable against a schema, and a person was already involved.

Drafting for review. The reply, the summary, the description. A human is the next step by design, so errors are caught by the workflow rather than by a customer.

Routing and triage. Deciding which queue, which team, which priority. Misroutes are recoverable, visible, and cheap. The alternative — a rules engine of two hundred keyword conditions nobody dares touch — is often worse.

Reconciling near-matches. Deciding whether "Acme Freight Ltd" and "ACME Freight Limited" are the same company. Fuzzy string matching gets most of the way; the model handles the residue, and the residue is where the human effort actually goes.

Four positions that do not

Arithmetic. Do not ask a model to total a column. Ask it to identify which numbers are the column, then total them in code. This sounds obvious and is one of the most common failures we see in inherited systems.

Authoritative lookup. "What is this customer's current balance" is a database query. A model that answers from context is answering from a copy of unknown freshness.

Anything irreversible. Sending, paying, deleting, granting. Not because the model is untrustworthy in general, but because the cost of the tail case is unbounded and the tail case is guaranteed to arrive eventually.

Deterministic policy. If a rule is written down and does not need judgement — refunds under a threshold are automatic, over it are not — implement the rule. A model asked to apply a fixed rule will apply it almost always, which is the worst possible reliability profile for a rule.

The shape that usually results

Once you sort a process this way, the architecture stops being "an AI feature" and becomes something more boring and much more robust: a pipeline where the model does interpretation at the boundaries, deterministic code does calculation and enforcement in the middle, and a person is positioned wherever the cost of being wrong exceeds the cost of a glance.

That system is less impressive in a demo. It is the one that is still running in a year.

A test you can run this week

Take the process you are considering. Write out its steps. Against each one, write the answer to "what happens when this is wrong, and who finds out?"

Any step where the honest answer is "we would not know" is not a step to automate with a language model until you have built the thing that would tell you.