Chapter 3 — Attention: The Idea That Changed Everything

At the end of the last chapter we left every word stranded on its own island. Each one had been placed on the map of meaning and stamped with its position, but each one’s meaning was still computed in isolation. The word “it” had no idea what it stood for. The word “bank” could not tell a river from a vault. To understand anything, the words must look at one another and revise themselves accordingly. The mechanism that lets them do this is called attention, and it is the breakthrough that made the modern era of AI possible. Almost every system in this book is built on it. So it is worth slowing down and getting the intuition exactly right.

The problem attention solves

Consider the sentence: “The trophy didn’t fit in the suitcase because it was too big.” What does “it” refer to — the trophy or the suitcase? You know instantly that it is the trophy, because trophies that are too big are the reason things don’t fit. Now change one word: "…because it was too small." Suddenly “it” is the suitcase. The meaning of “it” depends entirely on the other words around it, and on which of them turn out to be relevant.

This is the everyday situation of language. The meaning of a word is rarely self-contained; it is shaped by particular other words, sometimes far away. “The man who I met last Tuesday in Paris — he said…” The “he” reaches all the way back across the sentence to find “the man.” A model that processed each word in isolation would be lost. What we need is a mechanism by which each word can survey the whole sentence, decide which other words matter to it, and pull in information from exactly those words.

That is attention, in one sentence: a way for every word to look across the entire sentence, weigh how relevant each other word is to it, and update itself by drawing in the most relevant ones.

Figure 1. One word changes, and “it” points somewhere else entirely.

The party analogy

Here is a way to picture how attention actually decides what is relevant. Imagine the words of a sentence as people at a party, and imagine that each person carries three things.

Each person carries a question — a statement of what they are looking for. The word “it,” for instance, is looking for whatever noun it refers to.

Each person also wears a label — a short description of what they themselves are about. “Suitcase” wears a label saying, in effect, “I’m a container, a physical object, a thing that holds stuff.”

And each person carries some information — the actual content they can contribute if someone finds them relevant.

Now the mechanism runs. Each word takes its question and compares it against everyone else’s label, looking for strong matches. When “it” holds up its question and scans the room, the label on “suitcase” (or “trophy,” depending on the sentence) lights up as the best match. Having found its match, “it” pulls in the information that word carries, weighted by how strong the match was. Weak matches contribute little; strong matches contribute a lot. After this exchange, “it” is no longer a meaningless pronoun — it has absorbed the meaning of the thing it refers to. The islands have become connected. Every word, in parallel, does this same survey at the same time, so the whole sentence updates itself in one coordinated step.

This question-label-information dance is the entire idea. The technical names for the three things are query, key, and value, and you will meet them again, but the party picture is all you need to carry forward.

Figure 2. The party: a question, a label, and some information.

Why this was such a breakthrough

Before attention, the dominant approach processed text the way you might read with a finger on the page — strictly in order, carrying a running summary in your head, with older words fading as newer ones arrived. That made long-range connections fragile. By the time such a system reached “he” at the end of a long sentence, its memory of “the man” at the beginning had grown dim. These models struggled to hold a thought across distance.

Attention threw out the running summary and let every word reach directly to any other word, no matter how far away, in a single step. Distance stopped being an obstacle. A word at the end of a paragraph can attend to a word at the very beginning just as easily as to its immediate neighbor. This direct, any-to-any reach is what let models finally handle long, intricate, deeply nested language — and it is why the architecture built around attention, called the Transformer, displaced everything that came before it and now underlies essentially every system in this book.

Many conversations at once

There is a refinement that matters. A single round of attention can really only track one kind of relationship at a time. But a sentence has many relationships running through it simultaneously: who refers to whom, which adjective modifies which noun, what the overall topic is, what the emotional tone is. One survey of the room cannot capture all of that at once.

So the model runs many attention processes side by side, in parallel — picture not one party but several happening in the same room, each attended to with a different question in mind. One of these parallel processes might come to specialize in tracking grammatical agreement, another in matching pronouns to their nouns, another in following the topic. Each forms its own answers, and the model then combines them. Critically, nobody assigns these specialties in advance. They emerge on their own during training, simply because dividing the labor this way helps the model predict better. (These parallel processes are called attention heads, and you will see the count of them — “twelve heads,” “ninety-six heads” — quoted as a basic spec of a model.)

The hidden price

Attention’s great strength — that every word can look at every other word — is also the source of its central cost, and that cost shapes much of the engineering you will read about later. If every word must compare its question against every other word’s label, then the amount of work grows out of proportion to the length of the text. Double the length of a passage and you do not double the work; you roughly quadruple it, because every one of the now-twice-as-many words has to consider every other.

This is why processing long documents is expensive, why models have a “context window” beyond which they cannot see, and why a great deal of clever effort goes into making attention cheaper for long texts. Hold onto this fact. When later chapters talk about the cost of long conversations, or about tricks to handle book-length inputs, they are all wrestling with this same quadratic price tag that attention’s any-to-any reach demands.

Figure 3. Double the length of a passage and the work roughly quadruples.

What to carry forward

Attention is the mechanism by which words stop being islands. Each word poses a question, every word offers a label and some information, and each word updates itself by drawing in the information of the words whose labels best answer its question. Run many such conversations in parallel and you can track many kinds of relationship at once. This single idea — direct, any-to-any, learned relevance — is what made modern AI work, and it is the beating heart of every architecture in Part II. Its only real drawback is that it grows expensive as text grows long, a tension we will return to repeatedly.

We now have the three foundations: what a model is (Chapter 1), how words become numbers on a map of meaning (Chapter 2), and how those numbers learn to talk to each other (Chapter 3). With attention in hand, we are ready to meet the family of language models — four different ways of arranging this same machinery, each built for a different purpose. We begin with the most famous of them: the storytellers. Turn to Chapter 4 .

← Chapter 2 · From Words to Numbers · Contents · Chapter 4 · The Storytellers →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.