Chapter 10 — Raising a Helpful Assistant
At the end of the last chapter we had a fully trained model — fluent, knowledgeable, and, surprisingly, not very useful. This chapter is about the gap between those two things, and how it gets closed. It is one of the most important and least understood stages in the whole story, because it is where a raw text-predictor becomes the polite, helpful, well-behaved assistant people actually talk to. The crucial thing to hold onto throughout is that no new kind of learning appears here. Every stage below is the same downhill walk from Chapter 9. All that changes is the teaching material.
Why the raw model isn’t helpful
Imagine the result of all that training from Chapter 9: a model that has read a vast slice of the internet and become extremely good at predicting how text continues. This is called a base model, and it has a peculiar flaw. Ask it “What is the capital of France?” and it might not answer — it might instead continue with a list of more quiz questions, because on the web, a question like that often appears in exactly such a list. It is not being difficult; it is doing precisely what it was trained to do, which is to predict plausible continuations of text. It models how text behaves; it has not been taught to behave like an assistant.
The good news is that the helpful behaviors are already latent inside it — buried among everything else it absorbed, since the internet contains plenty of helpful answers too. They just need to be coaxed to the surface and made into the model’s default mode. That coaxing happens in two stages.
Figure 1. Ask a base model the capital of France.
Stage one: showing it good answers
The first stage is the most intuitive. You assemble a curated collection of example exchanges — a good question paired with an ideal answer, written or vetted by people — and you continue training the model on these polished demonstrations. It keeps doing exactly what it always did, predicting the next word, but now on examples of excellent, helpful responses rather than on the raw churn of the web. Importantly, it is graded only on the answer part of each example, not the question, so it learns the shape of being a good responder rather than how to ask questions.
This stage is often called supervised fine-tuning, and the analogy is an apprentice studying a portfolio of worked examples of excellent answers. Show enough of them and the model picks up the pattern: when someone asks something, you respond helpfully, directly, and in a consistent voice. This alone transforms the base model’s behavior — it now answers questions instead of echoing them.
There is a practical wrinkle worth mentioning, because you will hear its name. Adjusting all of a model’s billions of knobs for every bit of fine-tuning is expensive. A popular shortcut (called LoRA) freezes the original model entirely and learns only a small add-on adjustment — often less than one percent the size of the model — that snaps on top like a patch. It makes fine-tuning cheap and modular: you can keep one big frozen base model and maintain many small specialized patches for different purposes.
Stage two: learning from preferences
Showing good answers gets you far, but it hits a ceiling. For most real questions there is no single “ideal answer” to demonstrate. How long should a good explanation be? How should the model handle a sensitive topic, or gracefully decline a harmful request, or admit it doesn’t know? You cannot easily write the one perfect demonstration of these things. But there is something humans can do reliably even when they can’t write the best answer themselves: look at two answers and say which one is better.
This observation powers the second stage. The model’s answers are shown to people in pairs, and the people simply pick the better of each pair. From a large pile of these comparisons, a separate model is trained — a reward model — whose entire job is to predict human preference, to act as a learned, automated stand-in for human taste. Once trained, it can score any answer the way a person would, millions of times over, without a human in the loop.
Then the main model is tuned to produce answers that this reward model rates highly. Crucially, it is kept on a leash — not allowed to drift too far from the sensible language it already knew. The leash matters enormously, because without it the model discovers cheap tricks to game the scorer, producing strange text that the reward model loves but that reads to a human as gibberish. (This is a classic and instructive failure: optimize hard against any imperfect measure of “good,” and you get something that scores well and is bad.) With the leash holding it close to fluent language, this stage gives an assistant its characteristic feel — the polite tone, the sensible refusals, the calibrated hedging, the tendency to be genuinely helpful rather than merely plausible. This two-step dance of comparisons-then-tuning is usually called learning from human feedback.
There is a newer, simpler route that is increasingly popular, which skips building a separate reward model and the elaborate loop, and instead trains the main model directly on the human preference pairs — pushing it toward the answers people preferred and away from the ones they rejected. Same goal, less machinery. The details differ, but the spirit is identical: teach the model what people actually want, using comparisons rather than demonstrations.
Figure 2. Comparisons, a stand-in for human taste, and a leash.
The same pattern beyond chatbots
Although this two-stage story is told most often about chatbots, the underlying move — pretrain broadly, then adapt cheaply to a specific purpose — runs through the whole book. The reader models of Chapter 5 are adapted to specific tasks by attaching a small task-specific head and training briefly on a modest set of labeled examples, building on the general understanding they already absorbed. The mixture-of-experts models of Chapter 7 are fine-tuned with extra care, because their huge capacity makes them prone to over-memorizing small datasets — sometimes by freezing the router so a small dataset cannot tempt it back into using only a few experts. The translators of Chapter 6 are specialized on real pairs of source-and-target text. In every case the expensive, broad pretraining is done once, and adaptation is comparatively cheap.
The deepest contrast is between adapting a writer and adapting a reader. Adapting a writer is fundamentally a matter of behavior — coaxing a fluent generator into being helpful and well-mannered, which is what this whole chapter has described, and which can sometimes be achieved with nothing more than a cleverly worded prompt and no training at all. Adapting a reader is a matter of supervision — mapping its understanding onto specific labels, which always needs labeled examples. Steering generation versus mapping understanding: that is the cleanest summary of how the two great families are put to work.
The throughline
It is worth saying once more, because it is so easily lost: across every stage in this chapter, there is no new learning machinery. Raw web text, then curated demonstrations, then human preference judgments — the teaching material changes, but the engine is always the same downhill walk from Chapter 9. And once all of this shaping is finished and the model ships, it freezes, exactly as Chapter 1 promised. Your conversations with it do not nudge a single knob. The helpfulness was pressed into the model during these stages, before you ever met it.
Figure 3. One engine, three kinds of teaching material.
What to carry forward
A freshly trained model is a fluent predictor, not a helpful assistant — it models text rather than behaving like a respondent. It is raised into an assistant in two stages, both of them the ordinary learning of Chapter 9 pointed at different material. First it is shown curated examples of excellent answers, which teaches it the shape of being helpful. Then it is tuned against human preferences — usually via a learned stand-in for human taste, kept on a leash so it cannot game the score — which gives it its tone, its judgment, and its manners. The helpful behaviors were always latent in the base model; these stages bring them to the surface and make them the default. After that, the model is frozen and shipped.
The model is now both capable and helpful. All that remains is to understand what actually happens when you use it — the moment-to-moment business of generating an answer, and why that seemingly simple act is full of consequential choices and real costs. That is the final piece, in Chapter 11 .
← Chapter 9 · How a Model Learns · Contents · Chapter 11 · How a Model Thinks in Real Time →
The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.
