A model advertised as handling a million tokens is making two claims, and only one of them is usually true. The first is that it will accept an input that long without failing — a claim about position encodings, memory, and serving infrastructure, and one that is now routinely satisfied. The second is that it will use that input, attending to the relevant part of it as reliably at position nine hundred thousand as at position nine hundred. That claim is much weaker than the marketing implies, and the gap between the two is where most disappointment with long-context models lives. This piece is about what it takes to extend a window, what breaks when you do, and how to tell the difference between a model that can hold a document and one that can read it.
Three separate walls
It helps to separate the obstacles, because they were solved at different times by different means and are frequently conflated.
The memory wall was the first. Materializing the attention score matrix costs memory proportional to the square of the length, which put a hard ceiling on what could be run at any price. FlashAttention removed it by never writing the matrix down, dropping attention’s memory cost to linear. Note what this did not do: the time remains quadratic, so long contexts became possible without becoming cheap.
The position wall is that a model trained on four thousand tokens has never seen position four thousand and one. With learned position embeddings there is simply no entry in the table. With rotary embeddings there is a well-defined rotation, but it is one the model has never encountered, and performance degrades sharply past the training length. This is the wall that extension techniques address.
The serving wall is the KV cache , which grows linearly with length and is per-conversation rather than shared. This is what makes long contexts expensive in production rather than merely slow, and it is unaffected by anything the other two solutions do.
Only the second is really about the model. The other two are engineering, and they are the ones that got solved most convincingly.
Figure 1. Three walls, three different fates.
Extending the window
The standard route is to train on short sequences, which is cheap, and then extend, which is comparatively cheap too. Nobody pretrains at a million tokens; the quadratic cost makes it absurd.
The mechanism exploits a property of rotary embeddings. Position enters as a rotation whose angle is proportional to position, so extending the range means presenting the model with angles beyond anything it saw. Position interpolation rescales instead: divide every position index by some factor, so that a context four times longer maps onto the same range of angles the model was trained on. Positions that were one apart are now a quarter apart, which the model handles gracefully, having seen fractional-seeming relationships throughout training. The article on rotary embeddings describes this as re-spacing the seats so an oversized room looks familiar.
Plain interpolation has a flaw, and fixing it is where the refinements come in. RoPE uses many frequencies simultaneously — fast-rotating dimensions that distinguish nearby tokens, slow ones that encode long-range position. Interpolating all of them equally crushes the fast ones, degrading the model’s ability to tell adjacent tokens apart, which is a capability it uses constantly. The better schemes therefore rescale selectively: leave the high-frequency dimensions alone, since local relationships have not changed and those dimensions are not running out of range, and stretch only the low-frequency ones that actually need to cover new ground. Adjusting the base frequency of the rotation accomplishes much the same thing, and the two views turn out to be closely related.
With a good scheme, a few billion tokens of continued training on long documents — a fraction of a percent of pretraining — extends a model by ten or thirty times. This is why long-context versions of models appear so quickly after the originals. It is a modest post-hoc procedure, not a different model.
Attention dilutes
Here is the part that the extension techniques do not fix, and it follows from the softmax .
Attention distributes a fixed total of one across every position available. With a hundred candidates, an irrelevant one might receive a hundredth of the weight; with a hundred thousand, the same irrelevant position receives a hundred-thousandth, but there are now a thousand times as many of them. The aggregate weight sitting on material that does not matter grows with the length of the context, and the signal from the one passage that does matter has to compete against an ever-larger sum of noise.
This is not a bug in an implementation. It is what normalizing over more candidates does, and it means that a model’s ability to pick out a relevant fact from a haystack degrades as the haystack grows even when nothing else has changed. The effect compounds with a second one: models attend more reliably to the beginning and the end of their input than to the middle, a pattern robust enough to have its own name — lost in the middle. The beginning benefits from the attention-sink machinery and from being visible to every position; the end benefits from recency and from the local-attention heads that dominate the lower layers. Material placed halfway through a long document is attended to least, which has the entirely practical consequence that where you put something in a prompt affects whether the model uses it.
Figure 2. Dilution and the lost middle.
Measuring it honestly
The standard test is to hide a sentence in a long document and ask a question only that sentence answers. Models pass this at impressive lengths, and passing it means less than it appears.
The task is retrieval of a lexically distinctive string, and it is close to the easiest thing a long context can be asked to do. The planted sentence is usually semantically out of place, which makes it findable by surface features; only one fact must be located; and nothing has to be combined. A model can pass at a million tokens while being unable to answer a question requiring two facts from opposite ends of the same document.
The tests that discriminate ask for more. Multiple needles that must all be found and related. Aggregation across the whole input rather than a lookup — counting occurrences, tracking a variable through a long program, following a character’s state through a novel. Questions whose answers depend on material that is not lexically distinctive. Performance on these degrades much earlier than the single-needle result suggests, often falling off well before the advertised limit and sometimes before a quarter of it. A useful rule when evaluating a claim about context length is to ask what task was used to support it, because the answer is usually the easy one.
There is also a plain economic point that gets lost. Even where a long context works, it is not free: the cache cost scales with length, so a conversation carrying two hundred thousand tokens of history occupies memory that could have served many shorter ones. Filling a context because it is available is a real expense.
Figure 3. What the needle test hides.
Architectural alternatives
Two directions attack the cost rather than working around it.
Sparse and windowed attention restricts which positions may attend to which. Most heads see only a local window; a few keep global reach. Since a great deal of language is local, this loses less than one might expect, and interleaving local and global layers in a fixed ratio has become common in production models. It makes cost linear in length for most of the layers, and it is a genuine approximation — information more than a window away can only travel by hopping through intermediate positions across layers.
The more radical option is to abandon the growing cache entirely and carry a fixed-size summary of the past instead, which is what state space models do, at the cost of no longer being able to look anything up exactly. Hybrid designs that interleave both kinds of layer are currently the most credible attempt to get long-range capability without a cache that grows forever.
And there is the option of not extending the context at all. Retrieval puts a search step in front of the model and shows it only the relevant few thousand tokens, which is cheaper than a long context, easier to update, and easier to attribute — at the cost of depending on the retriever having found the right material. The two are often posed as competitors and are better read as answers to different questions: retrieval when the corpus is large and the query is focused, long context when the material is genuinely one document that must be considered as a whole.
What to carry forward
A long-context claim conflates two things: whether a model accepts the input, and whether it uses it. Three separate walls stood in the way, and they had different fates — the quadratic memory of the attention matrix, removed by never materializing it, though the time remains quadratic; the position encodings, which have no meaning past the training length; and the cache, which grows linearly per conversation and is what makes long contexts expensive to serve rather than merely slow. Only the second is really about the model, and it is addressed by interpolating positions so a longer sequence maps onto the range of rotations the model was trained on — selectively, leaving the high-frequency dimensions that distinguish nearby tokens alone and stretching only the slow ones — after which a small amount of continued training extends a window tenfold or more, which is why long-context variants appear so quickly. What extension does not fix is dilution: attention normalizes over every available position, so the aggregate weight on irrelevant material grows with the context, and models additionally attend less reliably to the middle of their input than to either end. Benchmarks obscure this, because locating one lexically distinctive planted sentence is nearly the easiest long-context task there is; tests requiring several facts to be combined, or aggregation over the whole input, degrade far earlier than the advertised limit. The alternatives are to restrict attention to local windows with a few global layers, to replace the growing cache with a fixed-size summary, or to retrieve the relevant few thousand tokens and skip the long context entirely — which is a different answer to a different question rather than a competitor.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
