Appendix D — Telling a Model Where Words Sit
Chapter 2 mentioned, almost in passing, that a model has to be told the order of its words. This appendix unpacks that, because the question of how to inform a model about word order turns out to have a small, elegant history of its own — and it explains one of the quiet reasons modern models can handle ever-longer texts.
Why order has to be spelled out at all
Start with a quirk that surprises people: the attention mechanism, left to itself, is blind to word order. It sees a sentence as a bag of words with no sense of sequence — so “dog bites man” and “man bites dog” look identical to it, the same words tossed in a bag. But those sentences mean opposite things, and the difference lives entirely in the order. Since order carries real meaning, and the core machinery ignores it, someone has to hand the model the seating chart explicitly. Every scheme below is just a different way of telling each word where it sits. They run from clumsy to elegant, and each was invented to fix a flaw in the one before.
Giving every word a seat number
The first idea is the obvious one: stamp each word with its seat number — first, second, third — and let the model learn what each seat tends to mean. This works fine and is easy. But it has a stubborn weakness. If the model only ever trained on texts up to, say, a thousand words, it has never seen a seat numbered one-thousand-and-one. Confronted with a longer text, it simply has no stamp for those later seats and falls apart. The seating chart was printed for a fixed-size room, and the model cannot cope when the room turns out to be bigger than any it trained in. That brittleness at unfamiliar lengths is the flaw everything afterward tries to repair.
Encoding the gaps, not the seats
The key realization is that what language actually cares about is rarely the absolute seat number — it is the distance between words. “The adjective two words before this noun”: that “two words before” is what matters, and it is true whether the phrase lands at the start of a sentence or the end. So instead of stamping each word with where it sits, you encode how far apart any two words are. This fits how language works much better, and it carries a lovely bonus: a distance like “three apart” or “ten apart” means the same thing no matter how long the overall text is, so a model built this way handles lengths it never trained on far more gracefully. It learned the gaps, not the seat numbers, and gaps do not run out.
Figure 1. Seat numbers run out; gaps do not.
Gently penalizing distance
One especially clean version of the relative idea barely “encodes” anything at all. It simply applies a soft penalty: the farther back a word is, the less weight it gets — attention quietly fades with distance, the way your memory of a conversation grows fuzzier the further back you reach. Nothing has to be learned or stored; it is a fixed, built-in lean toward recent words, with different attention heads set to fade at different rates so some stay tightly local while others reach further. Because it is just a simple function of distance with no fixed-size table behind it, a model using it can be trained on short texts and still run sensibly on much longer ones straight out of the box.
Rotating the words: today’s favorite
The modern winner is a genuinely clever maneuver. Picture each word’s representation as a little arrow on a dial. Before the words compare themselves to one another, you rotate each word’s arrow by an amount that depends on its position — the first word turned a little, the tenth word turned ten times as much, and so on. The magic is what happens when two rotated words then compare: because rotations simply add up, the comparison ends up depending only on the difference in their rotations — that is, on how far apart the two words are. So even though each word was rotated by its own absolute position, what comes out is purely about relative distance. You get the simplicity of stamping each word independently and the robustness of caring only about gaps, all in one move. This is why it became the default in most current models: it is clean, it costs nothing to store, and it naturally cares about the right thing. (This is the “modern trick” Chapter 2 alluded to.)
Figure 2. Rotate each word by its own position; only the gap survives.
Stretching a model to longer texts
There is a final, very practical bonus, and it is the reason this appendix matters beyond curiosity. A model trained on short texts with the rotation scheme can be coaxed into handling much longer ones by adjusting the dial settings. The intuition: if you trained in a small room and now face a much bigger one, you can gently compress the new seating so the far-apart words appear closer together — back inside the range of distances the model already understands — or you can selectively rescale the rotations so fine local detail stays sharp while only the long-range spacing gets stretched. With a touch of extra training, sometimes none at all, a model built for short texts can be expanded to read very long ones. It is effectively re-spacing the seats so an unfamiliar, oversized room looks like the familiar one the model grew up in. This is much of the machinery behind the headline “this model now supports a context of hundreds of thousands of words.”
The takeaway
Word order is meaning, but attention is blind to it, so order must be supplied. The history runs from stamping absolute seat numbers (simple but brittle at new lengths), to encoding the gaps between words (robust, because gaps don’t run out), to gently fading attention with distance, to the modern favorite of rotating each word by its position so that only relative distance survives the comparison. And because the best of these schemes care about distance rather than absolute position, they can be stretched — letting a model trained on short texts grow to read very long ones.
Figure 3. From seat numbers to rotations, each fixing the last.
← Appendix C · Attention Variants · Contents · Appendix E · The Chinchilla Story →
The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.
