The usual account of attention’s cost is that it grows with the square of the sequence length, and the well-known responses are aimed at that square. A machine serving a large model to a few hundred people at once is not usually failing on arithmetic. It is failing on space. Somewhere in its memory sits one key vector and one value vector for every token, of every head, in every layer, for every conversation in flight, and that structure — not the parameter count, not the score grid — decides how many people the machine can hold and therefore what a token costs to sell. The remedy has to be storing less per token. The technique that delivered it rests on an observation close to an insult: most of what is stored is redundant. This piece is about which part of an attention head carries its specialization, what it costs to make heads share the rest, and why the saving turns out to be so nearly free.
The bill lands at inference, not at training
The quadratic term is a training and prefill cost, since both process whole sequences at once and pay in full for the grid of scores. Generation does not work that way. A decode step handles one new token, whose query attends over everything stored so far, so the grid for that step is a single row. The square has collapsed to a line. The storage has not.
The key-value cache holds, for every token already read, one key and one value for each head of each layer, those being what future positions consult. It persists for the life of the request and grows by one entry per generated token. For a large model at a long context it runs to tens or hundreds of gigabytes, more than the weights beside it.
Now the asymmetry that makes it govern the economics rather than merely appear in them. The weights are read once per forward pass and used by every sequence in the batch, so sixty-four concurrent users amortize one pass over the weight matrices — the whole reason batching works, and why enough arithmetic sits idle for speculative decoding to pay. The cache amortizes over nobody; each sequence reads only its own. The weights are a fixed charge. The cache is per head.
So the binding constraint is how many caches fit beside the weights, and the chain from there is short: cache size sets batch size, batch size sets throughput, throughput sets the price. Which makes what follows an inference technique wearing an architecture’s clothing. It alters the definition of the model, must be settled before training, and is justified by a figure on an invoice.
The queries are where the specialization lives
A head, mechanically, is three projections. It takes the token’s current vector and produces a query saying what it looks for, a key advertising what it offers, and a value carrying what it hands over; attention runs many heads in parallel on slices of the representation, and they specialize without instruction. One tracks the preceding token, one matches an open bracket to its close, one carries a subject across a clause.
Ask where in those three the specialization is implemented. The query is the head’s question, and a head is defined by what it asks about. The key advertises properties the token possesses — a plural noun, an opened bracket, a concern with cooking — and those properties are nobody’s private business. The value is content, and content is content. Sixteen heads asking sixteen questions of the same noticeboard are already sixteen notions of relevance. The board does not need sixteen copies.
The claim is checkable rather than decorative. Keys produced by different heads in a layer turn out to be strongly correlated, and so do the values; a head’s output degrades far less when forced to read a neighbour’s keys than to ask a neighbour’s question. Strongly correlated is not identical, and that gap is where the quality loss comes from. The direction lands conveniently, though: the queries that carry the specialization are the one projection never worth storing, since a query is used once and discarded. The expensive part of a head is the part that matters least.
First one set for everybody, then eight
Multi-query attention takes the observation at full strength. Every head keeps its own query projection; the whole layer gets one key projection and one value projection, shared. Each head still asks its own question and produces its own output. The cache falls by exactly the head count — thirty-two heads, one thirty-second of the storage, a sixteen-gigabyte cache reduced to half a gigabyte.
Then the honest cost, measurable rather than theoretical. Quality dropped, modestly but detectably, and models trained this way from scratch were reported to be less stable during training. Neither problem was bad enough to abandon the idea, and both were bad enough that nobody wanted to ship it.
Grouped-query attention is the compromise that did ship. Divide the query heads into groups and give each group its own key and value projection: sixty-four heads as eight groups of eight. The cache falls by the ratio of groups to heads, one-eighth here, for a quality loss that careful evaluation finds and ordinary use does not. Essentially every model released since has been built this way.
It is not a new mechanism but an interpolation between two endpoints already in hand. The interesting part is the shape of the curve, which is remarkably flat. Sixty-four groups down to eight discards seven-eighths of the cache and costs little; eight down to one discards a further eighth and costs more than the seven-eighths did. The last saving is the expensive one, and nobody has a satisfying account of where the knee sits. The redundancy is real but not total, and eight is roughly where it runs out.
Uptraining is why this spread in months rather than years. A finished multi-head model can be converted rather than replaced: average the key projections of each group into one matrix, the values likewise, and the grouped architecture arrives with a serviceable initialization. Averaging blurs whatever the individual heads were doing differently, so the converted model is worse than its parent — and a short continuation of training, a few percent of the original budget, recovers most of the loss. Thirty-two near-identical photocopies become eight, and nobody had to gamble a from-scratch run on whether the trade was worth taking.
Figure 1. MHA, GQA and MQA, and what each caches.
Figure 2. The saving is nearly free until it is not.
A pinch of math: the ratio of groups to heads
The cache is what is being attacked, so write down its size. Per token, each layer stores one key and one value for every key-value head, each as wide as a head:
$$\text{numbers per token} = 2 \cdot L \cdot H \cdot d_h$$The $2$ is for keys and values, $L$ is the layer count, $H$ the number of heads, and $d_h$ the width of one head. Sharing replaces $H$ with the number of groups $G$:
$$\text{numbers per token} = 2 \cdot L \cdot G \cdot d_h$$so the ratio of the two is $G/H$ and nothing else; no other factor has moved. Put a large model in: eighty layers, sixty-four heads of width $128$, two bytes per number, about two and a half megabytes per token. At a context of a hundred and thirty thousand tokens, one conversation carries roughly three hundred and twenty gigabytes, more than any single accelerator holds. With eight groups it carries forty. With one group, five. Stack cache quantization on top and the factors multiply.
Now count the arithmetic. The query count has not changed: $H$ query heads still dot against every stored position, so the score grid is exactly $H$ by $n$ as before and the multiply-adds are identical. Each key-value pair is simply read by $H/G$ query heads instead of one. The same arithmetic, on one-eighth of the bytes — and since decode waits on memory rather than on the arithmetic units, that ratio decides how long it waits. One factor in a product moved and the flop count did not notice.
Figure 3. One conversation’s cache at 130,000 tokens.
What it does not fix
Three limits and one rival, stated plainly, because the technique is routinely credited with more than it does.
It saves memory, not computation. The grid of scores is the size it always was, so the quadratic term is untouched and prefill costs what it cost before: eight groups read a hundred-thousand-token prompt at the price of sixty-four. It does nothing for the compute side of long context , nothing that overlaps with the tiling in FlashAttention , and nothing for training. These compose; none substitutes for another.
It reduces expressiveness by construction, and the loss is small rather than zero. Eight key spaces exist where sixty-four did. Heads in a group can still ask different questions, but only of one shared set of keys, and a relationship needing a key space no sibling head wanted has nowhere to live. Losses of this kind average away in aggregate benchmarks and surface in the particular capability someone is relying on.
It interacts awkwardly with tensor parallelism . Splitting an attention layer across devices normally gives each device a subset of the heads. A group count below the device count means the key and value projections cannot be divided that way and must be replicated instead, along with the cache they produce, handing back part of the saving where memory was tightest. Eight groups across eight devices is clean; eight across sixteen is not, which is why group counts are chosen with serving topologies in mind.
A later line of work attacks the same constraint without sharing anything. Rather than forcing heads onto identical keys and values, it projects keys and values jointly into a small shared latent, caches the latent, and reconstructs what each head needs at attention time. Nothing must be equal, only to lie in a low-dimensional subspace, which recovers more quality per stored byte than sharing does. The cost is a considerably more complicated kernel. Both are in production at scale, and which is preferable is unsettled.
What to carry forward
Attention’s cost grows with the square of the length, but that square is a training and prefill cost; during generation the row of scores is one token wide and what fills the machine is the cache of one key and one value per token, per head, per layer, per conversation. Weights are read once per batch and shared by all of it; a cache is read by one sequence, so the cache rather than the parameter count sets how many users a machine holds, which sets throughput, which sets the price per token. The technique exploits an asymmetry inside the head: each head has its own query, key and value projection, the query carries the specialization, and keys and values are largely redundant across heads. So: a query per head, shared keys and values. Multi-query attention shares one key-value pair across all heads and divides the cache by the head count, at a cost of measurable quality loss and instability. Grouped-query attention interpolates — heads divided into groups, one key-value pair per group, commonly eight groups for sixty-four heads — keeping most of the memory saving and most of the quality; the notable fact is how flat that curve is, the last portion of the saving costing more than all the rest. An existing multi-head model can be converted by averaging the key and value projections within each group and training on a fraction of the original budget, hence the speed of adoption. It does not touch arithmetic: score grid and prefill cost are unchanged, expressiveness is reduced by a small but non-zero amount, and a group count below the device count forces the shared projections to be replicated rather than split. What it buys is concurrency in the one resource that was running out, and nothing else.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
