.jpg){kind=link}
Retrieval augmented generation has an intimidating name for a simple idea. Look things up first, then answer using what you found.
That is the whole concept. The reason it matters is that a language model knows what was in its training data and nothing about your company’s documents, your product catalogue or last Tuesday’s policy change. Retrieval is how you close that gap without retraining anything.
The shape of it
You take your documents and split them into pieces. You turn each piece into a vector, which is a list of numbers positioned so that pieces about similar topics sit near each other. You store those.
At question time you turn the question into a vector the same way, find the nearest pieces, and paste them into the prompt with an instruction along the lines of “answer using this material”.
That is it. The clever part is not the generation. It is the looking up.
Why it is usually the right first answer
Compared to fine tuning, retrieval is cheaper, faster to build, and much easier to keep current. When a document changes you re-index one document. There is no training run and no waiting.
It also gives you something fine tuning cannot: a citation. Because you know which pieces you retrieved, you can show the user where the answer came from. For anything where being wrong has consequences, that traceability matters more than a marginal quality gain.
Where it falls apart
Chunking decides everything and nobody wants to talk about it. Split too small and a piece loses the context that made it meaningful. Split too large and you retrieve a page of noise to get one relevant sentence. Most disappointing retrieval systems are disappointing here, not in the model.
Similarity is not relevance. Vector search finds text that is about the same topic. It does not find text that answers the question. Ask “which regions are excluded” and you may cheerfully retrieve three paragraphs about regions that are included, because they are topically almost identical.
Questions that need everything. “How many contracts mention indemnity” cannot be answered by fetching the five most similar chunks. Retrieval is built for questions with a local answer. Aggregate questions need a database, not a vector store, and pretending otherwise produces confident nonsense.
The negative answer problem. If the documents genuinely do not contain the answer, a model handed five loosely related chunks will often assemble something plausible from them. Getting a system to say it does not know is harder than getting it to answer, and it needs to be tested for explicitly.
What actually improves results
In rough order of how much difference they make in practice:
Better chunking, informed by how the documents are actually structured rather than a fixed character count. Hybrid search, combining vector similarity with old fashioned keyword matching, because keywords are unbeatable for names, codes and exact phrases. Reranking the retrieved set with a model that scores relevance properly rather than trusting nearest neighbour order. And rewriting the user’s question before searching, since people ask questions in ways that match nothing in the source material.
Notice that four of those five improvements are about retrieval, and none of them are about the language model. That is the usual result. When a RAG system gives bad answers, the failure is nearly always upstream of the generation.
The honest summary
Retrieval is the right default for questions over your own material. It is cheap, current, and it can show its working. It is also mostly a search engineering problem wearing an AI hat, and the teams that treat it that way get much better results than the teams that keep tuning the prompt.