A model’s knowledge is fixed at the end of training, distributed across billions of weights in a form nobody can inspect, edit, or attribute. It cannot tell you where it learned something. It cannot be told that a fact has changed without retraining. And when it does not know an answer it will frequently produce one anyway, because nothing in its training taught it to distinguish confident recall from confident invention. Retrieval-augmented generation addresses all of this by an intervention that is almost embarrassingly direct: before asking the model to answer, go and find the relevant documents, and put them in the prompt. Nearly all the difficulty, however, lives in the retrieval step and in the seam between the two systems, and the naive version fails in ways that are entirely predictable once that seam is understood. This piece is about where the pipeline actually breaks, what the seam demands of both halves, and when retrieval is the wrong instrument to reach for.
Parametric and non-parametric knowledge
The framing worth adopting is that a system of this kind has two kinds of memory.
Parametric knowledge is what the weights hold. It is fast, requires no lookup, generalizes across phrasings, and supports inference — the model can combine things it knows in ways nobody wrote down. It is also unverifiable, frozen at training time, impossible to edit reliably, and blends smoothly into fabrication at its edges.
Non-parametric knowledge is what sits in the context window. It is current as of retrieval, citable, editable by editing a document, and confined to whatever was actually fetched. It costs context, it costs the cache , and it is only as good as the search that produced it.
The two are complements. The model supplies language, reasoning, and the ability to synthesize; the retrieved documents supply the facts. The most common failure of intuition about these systems is to expect the retrieved documents to fix a reasoning problem, or to expect the model to fix a retrieval problem. Neither happens.
Figure 1. Parametric and non-parametric knowledge.
The pipeline, and where it actually breaks
The standard arrangement has four stages, and it is worth walking them in order because the failure modes distribute very unevenly.
Chunking. Documents are split into passages, because retrieving a whole book is useless and retrieving a sentence loses its context. This unglamorous step causes a startling share of real failures. Split too small and a passage lacks the context to be interpretable — a paragraph beginning “This does not apply to contractors” is worse than useless without knowing what this is. Split too large and a passage contains one relevant sentence surrounded by noise that dilutes both its retrieval score and the model’s attention. Splitting on structure rather than on a fixed token count helps, as does prepending each chunk with its document title and section heading so it can stand alone.
Indexing and retrieval. Each chunk is turned into a vector by an embedding model and stored; a query is embedded the same way and the nearest chunks are returned. This is the subject of its own article, on retrieval embeddings , and the essential point is that these are not the token embeddings inside a language model but a separate model trained specifically so that a question lands near its answer.
Reranking. The first-stage retrieval is fast and approximate, comparing precomputed vectors. A second stage takes the top fifty or hundred candidates and scores each against the query with a much more expensive model that reads both together — which is far more accurate, because it can attend across the two rather than comparing summaries of each. Reranking is the single highest-value addition to a naive pipeline and is frequently omitted.
Generation. The surviving passages go into the prompt with an instruction to answer from them.
The distribution of blame is the practical lesson. When these systems give bad answers, the cause is retrieval far more often than generation — the right passage was never fetched, or was fetched and ranked twentieth, or was split in half by the chunker. Debugging the prompt when the retrieval is broken is a common and expensive mistake.
Figure 2. Four stages, and where the blame actually falls.
The seam
Several problems arise specifically from joining a search system to a language model, and they have no analogue in either alone.
The query is not the question. A user asks “what about the second one?” and there is nothing in that string to search with. Conversational context has to be folded into the query before retrieval, usually by having the model rewrite it into a standalone form first. Related techniques go further: generate several alternative phrasings and retrieve for each, or have the model write a hypothetical answer and retrieve documents similar to that, on the reasoning that an answer resembles other answers more than a question does.
Position matters. As the article on long context describes, models attend less reliably to the middle of their input. Ordering retrieved passages by relevance and dumping them in sequence puts the fourth-best passage in the worst position. Placing the strongest material at the beginning and the end measurably improves answers, which is an unsatisfying thing to have to know but is worth knowing.
Retrieved text can be wrong, and the model may prefer it anyway. If a fetched passage contradicts what the model knows, the outcome depends on how confident the model is in its own parametric knowledge — and models are, on the whole, quite deferential to their context. That is usually the desired behaviour, since the context is the current information. It also means the retrieval system is a channel through which false or adversarial content reaches the answer, which is a genuine security consideration when documents come from anywhere untrusted.
Attribution is weaker than it looks. Providing sources is one of the main arguments for the whole approach, but a citation attached by the model is a claim about where the information came from, not a verified link. Models cite passages they did not use and omit ones they did. Systems that need real attribution enforce it structurally — constraining generation to quote, or checking each sentence against the sources afterwards — rather than trusting the model to report honestly.
When to retrieve at all
The naive pipeline retrieves on every query, which is wrong in both directions: it wastes context on questions the model can answer unaided, and it retrieves once when a multi-step question needs several rounds.
The refinement is to make retrieval a decision rather than a reflex. A model can be trained or prompted to emit a search when it needs one, judge whether the results are sufficient, and search again with a refined query if not. This turns retrieval into tool use — the search becomes one action among several available to a model that is deciding what to do next — and it is the direction essentially all serious systems have moved in. It handles the multi-hop case naturally, since finding the answer to the first part supplies the query for the second, which a single-shot pipeline cannot do at all.
Figure 3. Retrieval as a reflex, and retrieval as a decision.
Against the alternatives
Two comparisons come up constantly and both deserve a straight answer.
Versus a long context. Now that models accept hundreds of thousands of tokens, why not skip retrieval and supply everything? Sometimes you should. But the corpus is usually far larger than any window; the cache cost of a full context is paid on every request while an index is built once; and the dilution effect means a model given a hundred relevant passages and ten thousand irrelevant ones does worse than one given the ten that matter. Retrieval is a filter, and filtering remains valuable even when the container is large. The techniques also compose — retrieve broadly, then let a long context hold everything retrieved.
Versus fine-tuning. These address different problems and are constantly confused. Fine-tuning is for teaching a model a behaviour: a format, a tone, a domain’s idiom, a task structure. Retrieval is for supplying facts. Attempting to teach facts by fine-tuning works poorly — the model learns them unreliably, cannot be corrected without another round, and gains no ability to cite. Attempting to teach a format by retrieval works poorly for the mirror reason. The right combination for a specialized application is usually both: adapt the model for how to behave, retrieve for what is true.
What to carry forward
Retrieval-augmented generation separates a system’s knowledge into two kinds — parametric, held in the weights, which is fast and generalizes and cannot be inspected or updated, and non-parametric, placed in the context, which is current and citable and confined to what was fetched — and lets each do what it is good at. The pipeline chunks documents, indexes them as vectors, retrieves the nearest to a query, reranks the candidates with a more expensive model that reads query and passage together, and generates from the survivors; of these, chunking and reranking are the steps most often done badly, and the overwhelming majority of bad answers come from retrieval failures rather than generation failures. The seam between the two systems creates its own problems: a conversational query is not a searchable one and must be rewritten; models attend least well to the middle of a long input, so passage ordering matters; retrieved text overrides the model’s own knowledge, which is usually right and makes the retriever a channel for anything false that gets indexed; and citations produced by a model are claims rather than verified links, so real attribution has to be enforced structurally. The naive pipeline retrieves once per query, which is wrong for questions needing several hops and wasteful for those needing none — so serious systems make retrieval a decision the model takes, which turns it into a form of tool use. Against a long context it remains valuable as a filter, since irrelevant material dilutes attention and an index is built once while a context is paid for every request; against fine-tuning it is simply orthogonal, one supplying facts and the other behaviour.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
