Appendix C — A Field Guide to Attention Variants
Chapter 3 introduced attention as the meeting where every word glances around at the others and decides which ones matter. That one mechanism is the whole game — but in real models it comes in several outfits, and the names can pile up into intimidating alphabet soup. This appendix is the decoder ring. The trick is to see that every “attention variant” is just that same meeting run under different rules, and that the rules answer practical questions: what does the task require, and what can the hardware afford? It helps to picture three independent dials — who is allowed to look at whom, how much memory the looking costs while writing, and how to keep the looking affordable when the text gets very long. Let us turn each in turn.
Dial one: who is allowed to look at whom
The most important choice is not about saving money — it is forced by the job. There are three settings.
Looking only backward (causal). When a model is generating text one word at a time, each word may look only at the words before it, never after. The reason is almost obvious once said aloud: the model’s job is to predict the next word, and letting it peek at the future would be cheating, like a student seeing the answer key. So every word wears blinders blocking the road ahead. This backward-only style is the heart of every chatbot-style writer, and it is exactly the rule from Chapter 4 .
Looking both ways (bidirectional). Sometimes the whole text is already present and nothing needs to be generated — you just want it understood, as when classifying a review or building a search index. There is no future to protect, so every word may look in both directions and draw on everything around it. This is the style behind the reader models of Chapter 5 , and it is also how models read images that have been chopped into patches. It is wonderful for comprehension and useless for generation, because the moment a word can see the future, next-word prediction collapses.
Looking at a different text (cross-attention). The third setting has words in one text look over at a second text entirely — the mechanism of “pay attention to that other thing while you work,” and the one new idea behind the encoder–decoder models of Chapter 6 . The classic case is translation: as the model writes the French sentence, each French word glances back at the English original. The same trick lets a text model look at an image, so the words being generated can attend to what is actually in the picture. Modern all-in-one chatbots mostly skip it by stuffing everything into one long prompt, but it remains the natural tool whenever there are genuinely two separate sources to reconcile.
Figure 1. Dial one: three settings, and the task picks for you.
Dial two: sharing notes to save memory
The remaining variants exist purely to make attention cheaper, and the first cost problem appears while the model is writing. Recall from Chapter 3 that a model runs many attention meetings in parallel, called heads, each specializing in a different kind of relationship. Many heads help quality. The trouble is that, to avoid redoing work, the model keeps a running notebook of what each head has computed about every word so far (this is the cache from Chapter 11 ). With many independent heads, that is many separate notebooks, and as a conversation grows long the notebooks balloon until memory, not arithmetic, becomes the bottleneck.
The fix is delightfully simple: let the heads share notes. Instead of every head keeping its own private record of what each word is and offers, many heads consult one shared record. The most aggressive version has all heads share a single common notebook — multi-query attention — which shrinks memory dramatically but, by sharing so hard, blurs the heads’ individuality and dips quality a little. So the field settled on a middle ground, grouped-query attention, now the standard: picture the heads sorted into a few small teams, everyone on a team sharing one notebook but different teams keeping separate ones. You get almost all the memory savings of full sharing while keeping almost all the quality of full independence, which is why most modern open models use it. (A newer, fancier idea compresses the notebook into a tight summary and expands it on demand, pushing the savings further, but it is more complex and not yet everywhere.) The thing to remember: none of these tricks change what the model fundamentally computes; they shrink the temporary scratch memory used during generation, trading a sliver of quality for the ability to serve long conversations affordably.
Figure 2. Dial two: how many notebooks the heads keep.
Dial three: keeping it affordable on long texts
The second cost problem is different. Run attention in full and every word looks at every other word, so doubling the length of the text roughly quadruples the work — nothing for a paragraph, brutal for a whole book. This is the quadratic price tag flagged back in Chapter 3. Several tricks tame it.
Only look at nearby words (sliding window). The simplest idea: rather than surveying the entire text, each word looks only at a window of its recent neighbors — the way you can follow a conversation by tracking the last few sentences instead of re-reading the whole transcript. The cost now grows gently with length instead of exploding. And because the model is many layers deep, information still travels far: a word sees its neighbors, who saw their neighbors, so news ripples across the whole text over several layers even though no single layer looks far. The tradeoff is that truly long-range links get weaker, which is why designs often mix in occasional full-view layers.
Only look at a chosen few (sparse attention). A relative of the window: instead of looking at everything, each word looks at a carefully chosen sprinkling of others — some neighbors, plus a few “anchor” words everyone may consult. It is efficient and clever, but the patterns are hand-designed and a bit brittle, so it never became mainstream for chatbots.
The clever bookkeeping trick (FlashAttention). This one is worth highlighting because it is not an approximation. It computes exactly the same attention as the plain version — identical result, no corners cut — but far faster. The insight is purely about how computers handle memory. Naively, the model builds a giant table of every-word-against-every-word, parks it in slow memory, then reads it back, and that shuffling to and from slow memory is the real bottleneck, not the arithmetic. The trick does the work in small tiles right inside the chip’s fast scratch memory and never writes the giant table out at all. Same math, dramatically less memory traffic, much quicker. Its arrival was so effective that it largely removed the pressure to use the approximate shortcuts above — the field mostly returned to full, exact attention because smarter bookkeeping made it cheap enough.
Figure 3. Dial three: how much of the text each word actually sees.
The takeaway
Attention is one idea wearing several outfits. One dial sets who may look at whom, and that one is dictated by the task — backward-only for writing, both-ways for understanding, across for reconciling two sources. The other dials are about cost: heads sharing notebooks to shrink the memory used while generating, and windows, sparse patterns, or smarter bookkeeping to keep the looking affordable on long texts. None of it changes the soul of the mechanism from Chapter 3; it just makes that mechanism fit the job and the hardware.
← Appendix B · Comparison Cheat Sheets · Contents · Appendix D · Where Words Sit →
The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.
