The generation loop takes one line to state: the model reads everything so far, produces a distribution over the vocabulary, one token is chosen and appended, and the whole thing runs again. Described that way it sounds tidy, and it hides something absurd. Taken literally, producing the thousandth word of an answer would mean pushing all nine hundred and ninety-nine earlier words back up through every block of the tower, from scratch, to compute representations the model had already computed nine hundred and ninety-nine times before. No system does this, and the thing that saves it is the KV cache — a running set of notes that turns a re-reading problem into a memory problem. This piece is about what is in those notes, why they consist of keys and values and nothing else, and why the memory problem they create now governs almost everything about what it costs to run a model.
The waste in the obvious loop
Start by pricing the naive version, because the number is what motivates everything else.
To produce one new token, the model must run a forward pass. If that pass has to process the entire sequence, then generating a response of $N$ tokens costs a pass over one token, then over two, then over three, and so on — which totals on the order of $N^2/2$ token-positions of work. A two-thousand-word answer would require processing something like two million token-positions rather than two thousand. The model would spend a thousand times more effort re-deriving what it already knew than on the words it was actually producing.
Now the crucial observation, and it is a property of the architecture rather than an optimization anyone invented. In a decoder, attention is causally masked: every position may look backwards and none may look forwards. So the representation the model builds for token five depends on tokens one through five and nothing else. When token six arrives, token five’s representation does not change. It cannot change, because token five was never permitted to see token six in the first place.
That is what licenses caching. The recomputation is not merely expensive; it is provably redundant, producing bit-for-bit identical results every time. Whatever the model worked out about the earlier part of the sequence remains true, and the only genuinely new work is whatever the newest token requires. Caching here is not an approximation and costs nothing in quality — it is the removal of work that was pure waste.
Why keys and values, and not queries
So the model should save its work on earlier tokens. Save what, exactly? The answer is more specific than “the earlier computation,” and the specificity is the whole reason for the name.
Recall what each token contributes inside an attention head. From its own vector it produces three things: a query, expressing what it is looking for; a key, advertising what it offers to anyone looking; and a value, the content it hands over to whoever attends to it. Attention pairs each token’s query against every earlier token’s key, and mixes the corresponding values in proportion.
Now ask, for each of these three, whether it is ever needed again after its own position has been processed.
A key is. When token six arrives, its query is compared against the keys of tokens one through five — keys that were produced when those tokens were processed, and that will be consulted again by token seven, and token eight, and every token thereafter. The same is true of values: token five’s value will be mixed into the output of every future position that attends to it. Keys and values are read by other tokens, indefinitely into the future.
A query is not. Token five’s query was used exactly once, at position five, to gather from the past. Nothing in the future ever consults it, because attention flows backwards only. The moment position five is finished, its query is dead.
That asymmetry is the entire design. Cache the keys and the values; throw the queries away. Hence the name, and hence the shape: two tensors per attention layer, holding one key vector and one value vector for every token processed so far.
With those notes in hand the generation step becomes narrow. The new token enters at the bottom of the tower as a single vector. At each layer it produces its own query, key, and value; its query attends over the cached keys and values plus its own; the result goes through the rest of the block; and its freshly minted key and value are appended to the cache for that layer before the vector moves up. What travels through the tower is a column exactly one token wide, consulting a widening archive at every level. The cost per generated token no longer grows with the length of what came before — except in the attention step itself, which must still scan the whole cache, and which is cheap by comparison with everything else.
Figure 1. Recomputing the tower versus consulting the notes.
A pinch of math: how big the notes get
The cache is not free, and its size is worth being able to estimate on the back of an envelope, because it explains more about the economics of serving models than any other single number.
For each token, each layer stores one key and one value per attention head, and each of those is a vector as wide as a head. So:
$$\text{bytes per token} = 2 \times L \times n_{kv} \times d_{\text{head}} \times b$$The $2$ is for keys and values, $L$ is the number of layers, $n_{kv}$ the number of key-value heads, $d_{\text{head}}$ the width of each, and $b$ the bytes per number.
Put a real model in. Thirty-two layers, thirty-two heads of width one hundred and twenty-eight, stored in half precision at two bytes each. That gives $2 \times 32 \times 32 \times 128 \times 2$, which is a little over half a megabyte — per token.
Half a megabyte does not sound like much until you multiply. An eight-thousand-token conversation carries a four-gigabyte cache. A thirty-two-thousand-token one carries sixteen gigabytes, which for a seven-billion-parameter model in half precision is more memory than the model’s own weights. And these figures are for a single conversation. Serve thirty-two users at eight thousand tokens each and the caches alone want a hundred and twenty-eight gigabytes, against fourteen for the weights they are all sharing.
That last comparison is the one to hold onto, and it has a clean statement. The weights are a library: one copy, consulted by everybody. The cache is a desk: every reader needs their own, and it keeps growing as they work. Adding readers costs you nothing in library, and a desk apiece.
Figure 2. Library and desks — weights against caches.
Two regimes: filling the cache and using it
Serving a request divides into two phases that look superficially similar and behave like different machines.
Prefill is what happens to the prompt. The whole thing is available at once, so every position can be processed in parallel — the same computation the model performs during training. The matrices involved are large and dense, the hardware is doing what it is built for, and the phase is limited by raw arithmetic. It is compute-bound. Its output, besides the first generated token, is a fully populated cache.
Decode is everything after: one token at a time, each depending on the one before, with no way to parallelize across the sequence. And here the arithmetic goes strange. To produce a single token, every weight in the model must be read out of memory — all fourteen gigabytes of a seven-billion-parameter model — in order to perform roughly fourteen billion multiply-adds. That is about one arithmetic operation for every byte moved. A modern accelerator can perform several hundred operations in the time it takes to move one byte from its memory. So during decode the arithmetic units sit idle well over ninety-nine percent of the time, waiting for weights to arrive. The phase is memory-bandwidth-bound, and how fast a model generates text is, to a first approximation, a question of how fast its weights can be streamed rather than how fast the chip can compute.
This split explains why serving systems report two separate numbers — time to the first token, and time per token thereafter. They are governed by different physics, respond to different optimizations, and trade against each other. A long prompt with a short answer is a prefill problem; a short prompt with a long answer is a bandwidth problem.
Figure 3. Prefill is compute-bound, decode is bandwidth-bound.
Why batching is the whole business model
Follow the bandwidth argument one step further and something important falls out.
If decode is spending nearly all its time reading weights and almost none doing arithmetic, then the arithmetic is free. Process sixty-four sequences at once and you read each weight matrix once, then use it sixty-four times instead of once. The reading — the expensive part — is amortized across the whole batch. Sixty-four users can be served in barely more wall-clock time than one.
This is not a marginal efficiency. It is the reason serving a model costs cents rather than dollars, and it is why every inference system batches as aggressively as it possibly can. But notice what limits it. The weights amortize; the cache does not. Each additional sequence in the batch brings its own desk, and the desks are what fill the machine. The size of the KV cache therefore sets the batch size, the batch size sets the throughput, and the throughput sets the price per token. Every megabyte shaved off the cache converts more or less directly into another concurrent user, which is why the allocator handing out that memory — the subject of paged attention — turned out to matter as much as the cache itself.
There is a scheduling refinement worth knowing, because it follows from the same asymmetry. If a batch is assembled and run to completion together, every sequence is held hostage by the longest one, and finished slots sit idle. Continuous batching instead schedules at the level of individual decode steps, retiring a sequence the moment it ends and admitting a waiting request into the freed slot on the very next iteration. The batch is a revolving door rather than a bus, and in practice it is worth several times the throughput of the naive arrangement.
Making the notes smaller
Given that the cache determines the economics, a great deal of ingenuity has gone into shrinking it. The formula above has four factors that can be attacked, and all four have been.
The most widely deployed attack is on $n_{kv}$: let several query heads share one set of keys and values. Since queries are not cached, you can keep all thirty-two query heads — preserving most of the model’s ability to attend in different ways — while storing only eight sets of keys and values, or in the extreme only one. Eight groups instead of thirty-two divides the cache by four, for a quality cost small enough that essentially every recent model does it. It is a remarkably good trade, and it works precisely because the thing being shared is the part that has to be stored.
A subtler attack compresses the keys and values jointly. Rather than storing them directly, project them down into a much smaller shared vector, cache that, and reconstruct what is needed on the fly — the same observation about big matrices that underpins LoRA , that the useful content of a large object often occupies far fewer dimensions than the object has. Applied to the cache it can remove the great majority of it, and it is the approach behind the long contexts of some recent large models.
A third attack is on $b$, the bytes per number: store the cache in eight or even four bits rather than sixteen. This is quantization pointed at the cache instead of the weights, and it inherits the same crucial split — keep the stored form small, expand it back to full precision for the arithmetic. Keys turn out to tolerate this less gracefully than values, so the two are often quantized to different depths.
The last attack is on the sequence length itself, and it is the only one that is genuinely lossy. If old tokens can be evicted, the cache stops growing. Sliding-window attention keeps only the most recent few thousand positions; other schemes evict tokens that have attracted little attention historically. A surprising practical finding is that the very first tokens of a sequence must be kept regardless of content, because attention deposits a great deal of leftover probability mass on them, and discarding them destroys the model’s calibration in a way that discarding the middle does not. Whatever the policy, evicting means the model can no longer see what was evicted, so this is a change to behaviour rather than an optimization — the honest framing is that it trades recall of the distant past for the ability to run at all.
Managing the notes you keep
Alongside making the cache smaller sits the question of storing it well, and this turns out to be where a large fraction of real-world serving performance was hiding.
The obvious implementation allocates, for each request, one contiguous block of memory big enough for the longest response it might produce. It is also disastrous. Most requests finish far short of their limit, so most of every reservation is never used, and the leftover gaps are the wrong shape for the next request. Systems built this way commonly wasted more than half of the memory they had set aside.
The fix borrows an idea from operating systems, which solved this problem for physical memory decades ago. Break the cache into small fixed-size blocks — say sixteen tokens each — scatter them anywhere in memory, and keep a table mapping each sequence’s logical positions to wherever its blocks physically live. A sequence grows by claiming one more block when it needs one and no sooner. Fragmentation collapses to at most a few unused slots at the end of a single block, memory utilization goes from roughly half to nearly all of what is available, and since the cache determines batch size, that translates straight into throughput.
The page table then enables something better. If two requests begin with identical text — the same system prompt, the same few-shot examples, the same document being asked about repeatedly — the cache blocks for that shared prefix are also identical, and both requests can simply point at the same blocks. Nothing is copied until one of them diverges. In production this is enormous: a two-thousand-token system prompt prefixed to every request is prefilled once and shared by everyone thereafter. It is also the mechanism behind the prompt-caching features that model providers now expose and bill at a steep discount — the discount is real, because the work genuinely was not done again.
When memory still runs short, the remaining options are to move a cache out to system RAM and bring it back when the conversation resumes, or simply to discard it and recompute the prefill later. Which is cheaper is a live calculation rather than an obvious call, since prefill is fast and the link to system memory is slow, and a long-idle conversation is often more cheaply re-read than retrieved.
What this explains about everything else
The cache is one mechanism, but it sits underneath a surprising number of things that otherwise look unrelated.
Long context is a memory problem, not an intelligence problem. When a model is advertised as handling a million tokens, most of the engineering behind that claim is cache management — sharing, compression, eviction, paging — rather than anything about the model’s ability to reason over long text. The two are routinely conflated, and separating them makes a lot of product claims easier to read.
Speculative decoding works because of the bandwidth argument, and only because of it. A small fast model drafts several tokens; the large model then checks all of them in a single forward pass and accepts the longest correct prefix. This is a gain rather than a wash only because verifying five tokens costs the large model almost exactly what generating one costs — the weights are read once either way, and the extra arithmetic is free in a regime that had compute to spare. It is a way of converting idle arithmetic units into tokens, and the guarantee is exact: the accepted output is distributed identically to what the large model would have produced alone.
It sharpens the mixture-of-experts bargain. A sparse model already inverts the usual picture by spending memory to save computation. In a decode phase that was bandwidth-bound to begin with, the saved computation is worth less than it looks and the extra memory hurts more, since it competes with the caches for the space that determines batch size. Sparse models are still a good trade — but the case rests on serving them at large scale across many machines, where the routing spreads work that would otherwise be duplicated, rather than on the raw arithmetic saving.
And it is why thinking is expensive. A reasoning model that deliberates for ten thousand tokens before answering has built a ten-thousand-token cache, occupying a desk that whole time, before it says its first word to you. Test-time compute is paid for twice: once in generated tokens, and once in the memory occupied while generating them, which is memory not available to serve anyone else.
What to carry forward
Generating text one token at a time would mean re-deriving every earlier token’s representation on every step, and causal masking guarantees those re-derivations would be identical — so they are computed once and kept. What is kept is specifically the keys and values of each attention layer, because those are consulted by every future position, while a query is used once at its own position and then never again; that asymmetry is the whole reason for the name. The saved notes are large: roughly half a megabyte per token for a mid-sized model, which means a long conversation’s cache can exceed the model’s own weights, and unlike the weights it is not shared — the weights are a library, the cache is a desk per reader. Serving therefore splits into a compute-bound prefill over the prompt and a memory-bandwidth-bound decode that reads every weight to produce a single token, using well under one percent of the available arithmetic. That idle arithmetic is what makes batching nearly free and speculative decoding profitable, and it is what makes cache size the binding constraint: the cache sets the batch size, the batch size sets throughput, and throughput sets the price. Hence the effort spent shrinking it — sharing key-value heads across query heads, compressing keys and values into a low-rank latent, quantizing the stored form, and, at the cost of actual capability, evicting old tokens — and the equal effort spent managing it, with fixed-size paged blocks to kill fragmentation and shared blocks to let a common prefix be prefilled once for everybody, which is exactly what a prompt-caching discount is selling. Most of what sounds like a claim about long-context intelligence is, underneath, a claim about how well someone is managing this one data structure.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
