Chapter 4 — The Storytellers: Decoder Models
We now have our three foundations in hand: a model is a function set by learned knobs, text becomes points on a map of meaning, and attention lets those points talk to one another. From here on, we are simply arranging this machinery into different shapes for different jobs. The first and most famous shape is the decoder model — the kind of system behind ChatGPT, Claude, Llama, and Gemini. These are the storytellers: models built to write, one word at a time. This chapter is about how they work and what makes them so capable. (We will save the deep details of how they are trained and run for Part IV, where those topics get a full treatment across all model types. Here we focus on the architecture itself.)
A storyteller writes left to right
The defining trait of a decoder model is that it generates text the way you write a sentence: from left to right, one word at a time, each new word committed before the next is chosen. When the model is choosing its next word, it is allowed to look at everything it has written so far — but never at anything that comes after, because nothing after it exists yet. It is composing into the void.
This left-to-right rule has a name — causal attention — and a simple justification. Recall that these models are trained on the game of guessing the next word. That game only teaches anything if the model genuinely cannot see the answer. If, while guessing the word that comes next, it were allowed to peek at the word that actually comes next, the guess would be trivial cheating and the model would learn nothing. So during both training and use, each word’s attention is restricted to look only backward, at what has already been written. The storyteller writes blind to the future, exactly as a human author does.
Figure 1. Blinders on the road ahead.
How a tower of blocks refines meaning
A decoder model is not a single layer of attention. It is the same building block, stacked dozens of times — sometimes more than a hundred — into a tall tower. Your text enters at the bottom and rises through the stack, getting a little more refined and a little more context-aware at each level. Understanding what one of these blocks does, and why stacking them works, is most of understanding the whole machine.
Each block does two things in turn. First comes attention — the conversation among words from Chapter 3, where each word looks back over the text and pulls in what is relevant to it. This is the mixing step, where information flows sideways between words. Then comes a second stage in which each word, now holding the context it gathered, is processed on its own — transformed, refined, reconsidered in place. This second stage does not look at other words at all; it works on each word individually, and, perhaps surprisingly, it is where most of the model’s raw stored knowledge actually lives.
A clean way to picture the rhythm: attention is the group discussion where everyone shares notes, and the second stage is each person going off alone to think hard about what they just heard. Discussion, private reflection, discussion, private reflection — over and over, all the way up the tower. By the time a word reaches the top, it has been through dozens of rounds of gathering context and digesting it.
Researchers find that the layers loosely divide the labor among themselves. The early layers tend to handle surface features — basic grammar, word forms. The middle layers handle meaning and relationships. The late layers assemble the final answer. But, as always, nobody designs this division. It emerges on its own because it is a useful way to organize the work.
Two pieces of plumbing that make tall towers possible
Stacking dozens of layers sounds straightforward, but for years it was not. Deep towers tended to fail to learn, because the faint signal that tells each layer how to improve would fade to nothing as it traveled down through so many layers. Two unglamorous but essential pieces of plumbing fixed this, and they appear in every model in this book.
The first is the shortcut connection. Rather than each block replacing the information it received, it adds its contribution on top of what came in. This creates a clean highway running straight up through the entire tower, so information and the learning signal can travel the full height without getting lost. This single idea — let each layer add to a running stream rather than overwrite it — is what made very deep networks trainable at all. It is hard to overstate how important it turned out to be.
The second is normalization. As information climbs the tower, its raw scale can drift — growing too large or shrinking too small — and a deep stack of such drifts can spiral out of control. Normalization gently rescales the numbers at each step to keep them in a sane range, like a volume governor on a sound system that prevents runaway feedback. It keeps the deep tower stable enough to train and to run.
Figure 2. The tower of blocks, and the plumbing that keeps it standing.
Turning the top of the tower into a guess
After a word’s representation has risen through every block, it is rich, fully contextual, and ready — but it is still just an internal bundle of numbers, not yet a prediction. The final step, the output head, translates that bundle into a score for every possible next token in the model’s vocabulary. A high score means “this token is a likely continuation.”
Those raw scores are then passed through a step that converts them into clean probabilities adding up to one hundred percent. This step has a useful exaggerating quality: a token scoring a little higher than the rest ends up with a lot more of the probability. The model does not hedge limply across thousands of equally weak options; it commits, concentrating its confidence on a handful of plausible continuations. The result is a ranked set of odds: perhaps seventy percent on one word, eight percent on another, and a long thin tail of unlikely possibilities.
Notice what the model produces: not a word, but a distribution over possible words. Which word actually gets chosen from that distribution is a separate decision — and a surprisingly consequential one, full of dials that change the model’s whole personality. That choosing step is where a great deal of the texture of these systems lives, and we devote much of Chapter 11 to it.
Writing by repetition
Now put the pieces together to see how a storyteller actually produces a paragraph. You give it some text. It runs that text up through the tower and produces its odds for the next word. One word is chosen and glued onto the end. Then the whole, now-slightly-longer text is run through again, producing odds for the next next word, which is chosen and glued on. And again, and again. The essay, the answer, the poem, the working program — all of it is built one word at a time by this patient loop of predict, choose, append, repeat.
This is worth pausing on, because it is genuinely the trick. There is no grand plan formed in advance, no outline the model fills in. Coherence over a long passage emerges from the fact that each new word is chosen in full view of everything written so far. The model writes its way forward, and the structure holds because every step is conditioned on the whole growing context. A storyteller that has merely learned to choose each next word well, given everything before it, turns out to be able to write things that look very much like thought.
Figure 3. Predict, choose, append, repeat.
What to carry forward
A decoder model is a left-to-right storyteller. It writes one word at a time, each word allowed to attend only to what came before, because its whole education was the game of guessing the next word without peeking. Inside, your text rises through a tall tower of identical blocks, each block letting words gather context through attention and then digest it privately — kept stable and trainable by shortcut connections and normalization. At the top, an output head turns the result into odds over the next word, one word is chosen, and the loop runs again. From that humble repetition comes the full range of what these systems can do.
The decoder is built to write. But not every job is a writing job. Sometimes you do not want to generate text at all — you want to deeply understand a piece of text you already have. For that, the same machinery is arranged into a very different shape: a reader rather than a writer. That is Chapter 5 .
← Chapter 3 · Attention · Contents · Chapter 5 · The Readers →
The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.
