Take a freshly pretrained model — one that has read a large fraction of the written internet, that can finish a legal clause or a Haskell function with unnerving fluency — and ask it a plain question. What often comes back is not an answer. It is another question, or three more questions, or a table of contents, or the same question rephrased and then a different one. The natural reading is that the model is being obtuse, or does not know, or has been loaded wrong. None of those is what happened. Documents containing a question very frequently continue with further questions, which is what a list of frequently asked questions is, and the model is doing exactly the thing it was trained to do: guess what comes next. It is not withholding an answer. It has never been told that answering is the activity. This piece is about the stage that installs that convention, why it needs so little data, and the ceiling it cannot pass.
A pretrained model continues; it does not answer
The pretraining objective is one instruction repeated across trillions of tokens: given this prefix, predict the next token. Nothing in it mentions users, requests, turns, or helpfulness. To a model built that way, a question is a string with a distribution over what follows it — which, in scraped text, is not reliably an answer.
The mismatch is one of register, not knowledge, and there is a clean demonstration of it that predates the whole practice. Paste three worked examples into the prompt in a fixed shape — question, answer, question, answer — and the same base model completes the fourth pair correctly, often well. Nothing was added. The user supplied the format by hand and the capability walked straight out. This is few-shot prompting, and for a while it was the only way to get useful work out of a large model.
The gap is format, not capability. That is the spine of the whole stage: it explains why the fix is so cheap, and why it accomplishes less than it appears to.
Prompting one’s way around the problem works, and it costs more than it looks. The formatting is brittle, since a model conditioned on a pattern is conditioned on all of it, whitespace included, and the examples occupy context that could have held the problem. Worse, a continuer has no concept of a turn ending, so having answered it will cheerfully invent the next question and answer that too, and the output must be cut off by a heuristic rather than by the model deciding it has finished. A guest who has read every recipe in the kitchen and, asked what is for dinner, recites the titles: all the information, no use.
Figure 1. Continuation, few-shot, fine-tuned.
The mechanism is the same one, pointed at different data
Anyone expecting new machinery here is in for a disappointment. There is none. Supervised fine-tuning uses the identical next-token cross-entropy loss as pretraining and the identical optimizer , at a smaller learning rate for a handful of passes. Shown the training loop with the dataset removed, you could not tell which stage you were looking at.
What changes is the corpus. In place of scraped documents the model is fed prompt-response pairs: a request, and a response demonstrating how it should be met. The pairs are wrapped in a chat template, special tokens marking where a user turn begins and where the model’s turn ends, and that end-of-turn token is among the more important things taught here — it is how a continuer learns to stop.
One mechanical detail changes what is learned and is easy to miss. The loss is normally masked over the prompt. Prompt tokens stay in the context, fully visible to attention, so the response is conditioned on them exactly as at inference; but no gradient is taken on them. The model is graded only on what it was asked to produce.
The costs here are quiet and structural. Because the objective is unchanged, nothing in it separates a good demonstration from a mediocre one — the loss asks only that whatever sits in the dataset be made more likely, so every hedge, every needless preamble, every plausible-sounding error is fitted with the same enthusiasm as the substance. And because the update touches all the weights of a model that took months to pretrain, running too long or too hot degrades capabilities nobody was training on. Hence a short stage, a watched data mixture , and the popularity of parameter-efficient methods such as LoRA and its quantized variant QLoRA , which confine the update to a few added parameters and cap the damage.
A pinch of math: grading only the tokens the model was asked to write
Write a training example as a prompt $x$ and a response $y$ of length $|y|$. The objective is the ordinary one, restricted to the response:
$$\mathcal{L}(\theta) = -\sum_{t=1}^{|y|} \log p_\theta\big(y_t \mid x,\, y_{It is cleaner to write the same thing over the whole sequence $u_1 \ldots u_T$, with an indicator switching the loss off on prompt positions:
$$\mathcal{L}(\theta) = -\sum_{t=1}^{T} m_t \log p_\theta\big(u_t \mid u_{One choice inside this is genuinely unsettled. Dividing each example’s loss by its own count of response tokens weights every example equally whatever its length; summing across a batch and dividing by the batch total weights long responses more heavily. Implementations differ, the difference is rarely documented, and it is a plausible source of irreproducible results.
Figure 2. The mask, position by position.
Why a few thousand examples go so far, and what they cost
Pretraining consumes trillions of tokens; this stage often runs on a few thousand examples. The disparity is evidence.
Around 2023 several groups converged on the same uncomfortable finding: a curated set of roughly a thousand demonstrations, written by people who knew the domain, could beat hundreds of thousands of automatically generated ones. Diversity of request type mattered, consistency of response mattered enormously, volume far less than anyone expected. Read that as a measurement rather than a trick. If a thousand examples suffice, nothing substantial about the world is being learned here. What is installed is a mapping from a kind of request to a register — a length, a stance, a level of formality, a habit of addressing the question asked — over knowledge already complete.
Three costs travel with this, and none is solved by scale.
Demonstrations are slow and expensive. A good response to a technical question takes a competent person a long time to write, cannot be crowdsourced cheaply without quality collapsing, and does not amortize the way a scrape does.
Quality is bounded by the demonstrator. An annotator cannot write a good answer to a question they cannot answer. Where the model already exceeds its demonstrators — long arithmetic, obscure trivia, unusual languages — training on their attempts pulls it down rather than up, and the harder the domain, the thinner the supply of people who can set a ceiling above it.
Format is learned faster than judgement. The stylistic surface — the confident opening, the structured explanation, the calibrated-sounding caveat — is a shallow pattern present in every example, fitted within the first few hundred steps. Whatever discernment the demonstrations encode is sparse and specific by comparison. So a model sounds like a competent assistant well before it behaves like one, and a fluent wrong answer is much harder to catch than an incoherent one. A good deal of the difficulty in evaluating fine-tuned models starts here.
Figure 3. Register is fitted long before judgement.
Imitation has a ceiling built into it
The limit is not incidental to the method. It is the method.
Fine-tuning can only raise the probability of text it has been shown, so it can show a model what a good answer looks like and nothing else. It cannot say that answer A is better than answer B, because the loss has no way to represent a comparison. It cannot reward an answer nobody wrote, because there is no gradient for text absent from the dataset. It cannot express degrees of approval at all: an adequate response and an outstanding one, both in the file, are pushed up alike.
That ceiling is why the rest of post-training exists. Preference methods, whether the reinforcement learning route through PPO or the direct supervised reformulation of DPO , carry the one bit this stage structurally cannot: of two candidates, this one was preferred. Verifiable rewards go further, rewarding a correct answer no human ever wrote down, which imitation cannot do by construction.
A second, subtler failure is worth naming. During training the model predicts each token given a human-written prefix; during generation it predicts given its own. The distributions are not the same, and errors compound: one awkward token puts the model in a context slightly unlike anything it trained on, where its next prediction is slightly worse. This is exposure bias, an old observation about teacher forcing that never fully went away, and part of why decoding matters as much as it does. It also explains a mild irony: the best demonstrations are increasingly written by a stronger model rather than a person, which lifts the demonstrator ceiling and narrows the mismatch at once, while turning the stage into distillation and making the ceiling somebody else’s model.
What to carry forward
A pretrained model is a text continuer, so asking one a question returns a continuation of a document containing that question — another question, a table of contents, a rephrasing — because nothing in the pretraining objective ever mentioned answering. The deficit is format, not capability, as few-shot prompting showed years before anyone fine-tuned for it: supply the shape by hand and the ability walks out. Supervised fine-tuning installs that shape permanently, with no new machinery whatsoever — the same next-token cross-entropy , the same optimizer , a smaller learning rate, and curated prompt-response pairs in place of scraped text. The one mechanical wrinkle is masking: the loss is switched off over the prompt, still visible to attention but not graded, so the model neither spends capacity generating requests nor gets weighted by how verbose they were. It works on thousands of examples rather than billions because the knowledge was already there and only a mapping from request to register is being added — a reading supported by the finding that a thousand careful demonstrations can beat vastly larger noisy sets. The costs are that demonstrations are slow and expensive, that their quality is capped by whoever wrote them since nobody can demonstrate an answer they do not know, and that the surface register is fitted long before any judgement is, producing models that sound reliable well before they are. Above all, imitation cannot exceed what it imitates: nothing in this objective can say that one response is better than another, or reward a good answer nobody wrote, and every example in the file is equally worth copying. Preference learning and verifiable rewards exist because of that limit — and because a model trained always on human-written prefixes must, in the end, generate from its own.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
