Attention is the mechanism everything else in a transformer is arranged around, and it is almost always invoked as though it were already understood — named, granted, and moved past. It repays being paid for properly. The mechanism is not complicated — it is three linear projections, a dot product, a softmax, and a weighted sum, and you could write it on the back of a napkin. What is remarkable is how much follows from it: that a model can resolve a pronoun forty words back, that it can follow an instruction given at the top of a page, that the whole architecture parallelizes well enough to be worth training on a trillion tokens. This piece is about the mechanism itself, the reasoning that produced it, and the two things it conspicuously cannot do on its own.
The problem with passing a summary along
Before attention, the standard way to process a sequence was to walk along it, maintaining a running summary. Read a word, update the summary, move on. The summary was a fixed-size vector, and by the end of the sentence it was supposed to contain everything worth knowing about the sentence.
The trouble is visible as soon as you state it. Everything the model knows about the first word must survive, intact and undamaged, through every subsequent update, squeezed alongside everything it has learned since, in a vector that never grows. A sentence of forty words asks that same fixed budget to hold forty words’ worth of detail. Information from the distant past does not so much get forgotten as get crowded out.
The failure is worst exactly where it matters most. Consider a sentence like the trophy would not fit in the suitcase because it was too large. Resolving it requires comparing two candidates that appeared some distance back, and knowing something about trophies and suitcases. A running summary has to have anticipated that this comparison would be needed and kept the relevant material alive on the off chance. It cannot go back and look.
There is also a mechanical objection. A running summary must be computed in order — you cannot update the summary for position ten until you have the summary for position nine. That is a hard constraint on how much of the work can happen at once, and on modern hardware, which is built almost entirely around doing many things simultaneously, it is close to disqualifying.
Attention resolves both objections with the same move. Rather than compressing the past into a summary and consulting the summary, let every position look directly at every other position and pull out what it needs. Nothing has to be anticipated, because nothing was discarded. And since each position’s lookup is independent of the others’, they can all happen at once.
Three roles for every word
The question is how a position decides which other positions are worth listening to, and how much of each to take. The answer is that every token plays three roles at once, and each role gets its own vector, produced by multiplying the token’s current representation by a learned matrix.
Picture a noticeboard. Each token pins up a small advertisement describing what it has to offer — that it is a plural noun, that it is the subject of the sentence, that it concerns cooking. That advertisement is the token’s key. Each token also brings the goods themselves, the content it will hand over to anyone who takes it up: this is its value. And when a token’s turn comes to gather information, it posts a want-ad describing what it is looking for — a preceding noun, an unresolved referent, the object of this verb. That is its query.
The mechanism is then exactly what the picture suggests. Compare a token’s query against every key on the board. Where a query and a key match well, that token’s value is taken in large measure; where they match poorly, it is taken in small measure or not at all. What arrives back is a blend of values, weighted by how well each advertisement answered the request.
Three points about this are worth making explicit, because each is doing real work.
The comparison is a dot product, which is to say a measure of alignment between two directions — the same operation, and the same geometric reading, as the one that turns a final state into logits . Nothing is being matched symbolically; a query and a key “match” when they point the same way in a space the model arranged for itself.
The three roles are genuinely separate. What a token advertises need not resemble what it hands over. A token can announce “I am a date” while the value it supplies is the date itself; the advertisement is for finding, the value is for using, and separating them lets the model be found on one basis and useful on another.
And the query is asked from the token’s current state, not from the raw word. By the time a token reaches the tenth block, its representation already encodes a great deal of context, so the question it asks at layer ten is a far more sophisticated question than the one it asked at layer one. Attention is applied repeatedly, and each application asks better questions than the last.
Figure 1. Query, key and value on a noticeboard.
A pinch of math: scores, scaling, and the mask
Stack the queries for all positions into a matrix $Q$, the keys into $K$, and the values into $V$. Then the whole mechanism is one line:
$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V$$Take it piece by piece. $QK^\top$ is every query dotted against every key, producing a square grid of scores with one row per asking position and one column per answering position. The softmax turns each row into a set of weights that sum to one, so every position distributes a fixed total of attention across the candidates available to it. Multiplying by $V$ then takes the corresponding weighted blend of values.
Two details in that line deserve explanation, because they are the parts nobody guesses.
The division by $\sqrt{d_k}$, where $d_k$ is the width of a key, is a variance correction. A dot product of two vectors of length $d_k$ is a sum of $d_k$ terms, and the spread of such a sum grows with $\sqrt{d_k}$. Without the correction, wider keys produce systematically larger scores, the softmax receives inputs spread over a large range, and it saturates — one weight goes to essentially one and the rest to essentially zero, which both destroys the blending and flattens the gradient to nothing. Dividing by $\sqrt{d_k}$ keeps the scores in a sane range regardless of width. It is a small thing that the architecture does not work without.
The $M$ is the mask, and it is how the same equation serves different purposes. It is a grid added to the scores before the softmax, containing zero where attention is permitted and negative infinity where it is not — and negative infinity, exponentiated, is exactly zero, so a forbidden connection receives no weight at all rather than merely a small one. For a model that writes, the mask forbids every position from looking forward, which is what makes it legitimate to train on all positions of a sequence simultaneously: each position is predicting its successor without having seen it. For a model that reads, the mask is empty and every position sees every other. The difference between an architecture that generates and one that comprehends is, at this level, one triangular grid of negative infinities.
Figure 2. One triangular grid of negative infinities.
Many heads, many conversations at once
One set of queries, keys, and values would force the model to express every kind of relationship through a single comparison. A pronoun’s referent, a verb’s subject, a quoted phrase’s opening mark, the topic of the paragraph — all of these would have to be found by the same yardstick, and a query that is good at one is unlikely to be good at another.
So the mechanism is run several times in parallel, on different slices of the representation. The vector is divided into equal parts, each part gets its own query, key, and value projections, each part performs the whole procedure independently, and the results are concatenated and passed through one final matrix that mixes them back together. These parallel copies are the heads.
Crucially the total work is unchanged: sixteen heads each a sixteenth as wide cost what one full-width head would cost. What is bought is not capacity but independence — sixteen different notions of relevance, applied simultaneously, each free to specialize. And they do specialize, without instruction. Examine a trained model and you will find heads that attend almost entirely to the immediately preceding token, heads that track matching brackets, heads that carry a subject forward across clauses, heads that appear to do nothing identifiable at all. Nobody assigned these jobs. They are what the division of labour settled into.
The price: everything looks at everything
The elegance has a bill attached, and it is the defining constraint on the whole architecture.
That grid of scores has one entry for every pair of positions. A sequence of length $n$ therefore requires $n^2$ comparisons, per head, per layer. Doubling the length of the input quadruples the attention work. At a thousand tokens this is unremarkable — the rest of the network dominates. At a hundred thousand tokens it is the entire cost of the model, and the score grid alone, if it were ever written down in full, would occupy more memory than the model’s weights.
Nearly everything that follows in the practical literature is a response to this one fact: computing attention without materializing the grid , restricting which positions may see which , sharing keys and values between heads so that fewer need storing, or replacing the mechanism with something cheaper that behaves approximately the same. The quadratic term is the reason there is a field of research here at all rather than a solved problem.
Figure 3. Where the quadratic term takes over.
Two things attention does not do
It is worth ending on the mechanism’s honest omissions, because both are supplied elsewhere and both are easy to assume are included.
The first is order. Look again at the equation and notice that nothing in it refers to position. Attention compares every query against every key on equal terms, and if the tokens of a sentence were shuffled the set of comparisons would be identical. Attention is entirely blind to word order — a startling property for a mechanism used to model language — and position must be injected separately , most often by rotary embeddings .
The second is memory of its own work. Each application of attention consults the keys and values available to it and produces an output; it has no notion that these keys and values might be worth keeping. During training that does not matter, since the whole sequence is present at once. During generation it matters enormously, and the machinery for retaining them is the KV cache — which, note, is a cache of exactly two of the three roles, for reasons that the separation described above makes obvious.
What to carry forward
Attention replaces a running summary with direct access: rather than compressing the past into a fixed-size state and hoping the right things survived, every position reaches back and takes what it needs from every other, which removes the bottleneck and, because the lookups are independent, allows the whole sequence to be processed at once. Each token plays three roles, each a learned projection of its current state — a query saying what it is looking for, a key advertising what it offers, and a value carrying what it hands over — and the separation of key from value is what lets a token be found on one basis and be useful on another. Scores are dot products between queries and keys, divided by the square root of the key width to stop the softmax saturating as the model gets wider, and shifted by a mask of negative infinities that decides who may see whom: triangular for a model that writes, empty for one that reads. The procedure is run in parallel across several heads on slices of the representation, which costs nothing extra and buys independent notions of relevance that specialize on their own. The price is that every pair of positions must be compared, so cost grows with the square of the length — the single constraint from which most of the practical engineering around transformers descends. And two things are not included: attention is blind to word order, which must be supplied separately, and it has no memory of the keys and values it produced, which must be cached explicitly when generating one token at a time.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
