RAG is a retrieval problem wearing a language model costume

A team builds a system that answers questions over their documentation. It works in testing. It ships. Support starts forwarding answers that are confidently, specifically wrong.

The instinct is to change the model, or the prompt, or the temperature. Then someone checks what was actually retrieved for the failing question, and the answer is obvious: the model was handed three chunks that did not contain the answer, and it did what a language model does — produced a fluent response from what it had.

Almost every disappointing RAG system is a retrieval system that has not been evaluated as one.

Evaluate retrieval separately, first

Before measuring answer quality, measure whether the right document reached the model at all. Take a set of real questions. For each, record which document actually contains the answer. Then ask one thing of your retriever: is that document in the top k?

This number is usually much worse than people expect, and it is a hard ceiling. If the correct chunk is retrieved 60% of the time, no prompt engineering gets answer accuracy meaningfully above 60%. Work spent on the generation half while retrieval sits at 60% is work spent on the wrong half.

Where retrieval actually breaks

Chunking severs the answer. A fixed 500-token window splits a table from its header, a definition from its term, a step from its procedure. The chunk that gets retrieved is real and useless. Chunking on document structure — sections, rows, list items — beats chunking on length almost every time, and requires actually parsing the source format.

Embeddings miss exact terms. Vector similarity is good at meaning and indifferent to specificity. A query for error code E4021 or part number MX-88 retrieves documents that are semantically about errors and part numbers. Hybrid search — keyword and vector together — fixes a class of failure that no amount of embedding model upgrading will.

The corpus contains contradictions. Three versions of the same policy from three years. Retrieval faithfully returns the oldest. The model has no way to know it is stale, because staleness is metadata, not content. This is a data curation problem that looks like a model problem.

Queries are not documents. A user asks "why did my thing break". The document says "diagnosing intermittent connection failures". These are not similar strings and are only somewhat similar vectors. Query rewriting — expanding the user's question before searching — often buys more than any other single change.

What to do instead of upgrading the model

Build a retrieval test set. Fifty real questions, each labelled with the document that answers it. Report recall at 5 and at 10. Put it in CI. This is a day of work and it will redirect your effort permanently.

Chunk on structure, not length. Parse the format. Keep tables whole. Carry section headings into the chunk so context survives.

Use hybrid search by default. Vector plus keyword, then combine. The added complexity is small and it covers a failure mode that is otherwise invisible.

Attach metadata and filter on it. Version, date, product, audience. Then filter before ranking. Most "the answer was out of date" failures are a missing WHERE clause.

Make citation mandatory. Require the model to cite which retrieved chunk supports each claim, and verify the citation points at something real. This does not stop a wrong answer, but it converts an invisible failure into a checkable one — and it gives the reader a way to disagree with the system.

The reframe

Stop calling it an AI project. It is a search project with a generation step at the end, and search is a discipline with decades of evaluation practice behind it.

Teams who make that switch usually find their next three improvements were sitting in plain sight, in the retrieval layer, where nobody had thought to look.