Plot the attention weights of a trained transformer and the picture is nearly empty. A bright diagonal, a stripe down the first column or two, the occasional long arc joining a quotation to its opening mark — and around all of it a grey field of numbers too small to move anything. The grid that produced that picture was computed in full, at every layer and every head, one entry for every pair of positions, so the overwhelming majority of the arithmetic went into establishing that two tokens had nothing to do with one another. The obvious response is to stop paying for those: decide ahead of time which pairs are worth comparing, and skip the rest. That move is cheap to describe, has been tried in more variations than almost anything else in this area, and carries a catch that no amount of ingenuity has removed. This piece is about the schemes that restrict who may see whom, the bet each of them makes, and why the hardware has been so unfriendly to the elegant ones.
The grid mostly funds comparisons that come to nothing
The defining cost of attention is that the score grid holds one entry per ordered pair of positions, so comparisons grow with the square of the length, per head, per layer. At a thousand tokens nobody notices. At a hundred thousand it is the whole model.
Set against that is what the grid contains once training has finished. The weights come out of a softmax , whose exponential is deliberately sharp, and the distributions it produces are strikingly concentrated: a small fraction of the candidates carries nearly all the mass, and zeroing the rest afterwards would change the output by an amount lost in arithmetic noise. Most of the grid, in other words, funds comparisons whose answer turns out to be not relevant, and paying for an answer you were always going to discard is the definition of waste.
There is a difficulty inside that sentence, and it is the whole subject. A weight is known to be negligible because it was computed. Deciding which pairs matter is exactly what scoring does, so a scheme that declines to score a pair must decide without looking. This is a bet, not an optimization. The pattern is chosen at design time, before training and long before any particular input exists, and it commits every query at a given position to a fixed set of candidates. The arithmetic saved is guaranteed and quotable in advance. The information lost depends on the input, hides inside average loss, and appears only when the pair that was cut was the one that mattered. The saving is certain; the damage is occasional. That asymmetry is what makes the approach attractive and hard to evaluate.
A window is cheap, and information must then be relayed
The simplest pattern lets each position see only a fixed number of its immediate predecessors — a sliding window, or local attention. Fix the width and every query compares against that many keys however long the input, so cost becomes linear. Much of what language needs is local anyway.
The compensation relied on for everything outside the window is depth. A position sees its own neighbourhood at the first layer, but each neighbour has absorbed its neighbourhood, so the reach roughly doubles at the second and grows as layers accumulate — the argument made for a stack of convolutions, and arithmetically sound: with enough layers, any position can in principle influence any other.
The weakness is in what influence means here. Under a full grid, a position needing a fact from eight hundred tokens back fetches it in one hop from the position holding it. Under a window it cannot. The fact must be relayed — written into an intermediate position that is also mixing in everything else inside its own window, and passing along one vector of fixed width — then relayed again, each hop re-compressing more material into a representation that does not grow. Distant information is not forbidden so much as crowded out, which is the fixed-size-summary bottleneck attention was introduced to remove, reintroduced one layer at a time.
A pure window also breaks for a reason unrelated to reach. Run a model over a long stream, keep only the recent window, and the earliest tokens fall out of view. Quality does not degrade gently at that point; it collapses. Trained models use the first few positions to park attention that has nowhere useful to go, since a head with nothing to do must still distribute a full unit of weight somewhere, for reasons the softmax article sets out. Evict them and every idle head redistributes that weight onto real content and acts on it. The repair is effective and unglamorous: pin a handful of initial tokens in the cache permanently, beside the moving window. That is now standard in streaming systems, and it was found by accident.
Fixed patterns buy a shortcut, and one arrangement survived
The first influential formulations, from around 2019, added structure on top of the window, and their ingredients outlasted the particular designs.
Strided and dilated schemes buy reach cheaply by spacing the candidates out. A strided head attends to every eighth position rather than the last eight; a dilated window keeps the same count and spreads it with gaps, and if the gaps widen with depth the reachable span grows geometrically rather than linearly. The cost is resolution. A pattern with holes can reach a distant region but not resolve which token inside it was wanted — acceptable when the target is a topic, useless when it is a name.
Global tokens go the other way: designate a few positions that everything may attend to and which may attend to everything. Long-range information then has a cheap path: write to a hub, and anyone may read it. What that path is not is wide. Everything travelling beyond a window funnels through a handful of vectors, so the hub is a bottleneck by construction, and hubs must be present during training to be worth anything.
What survived into production is blunter than any of this. Interleave the layers: make most of them windowed and leave every fourth or sixth as ordinary full attention, so the cheap layers carry local structure and the bulk of the depth while the few expensive ones can look anywhere, directly, with no relay. It is the compromise reached by hybrids that replace the mechanism in most layers and keep exact attention in a few. Serving benefits disproportionately: windowed layers retain only a window of keys and values rather than the whole history, which shrinks the cache that limits how many conversations a machine can hold. The honest accounting is that this does not remove the quadratic term. It divides it by the fraction of layers that keep it.
Figure 1. Four patterns over the same score grid.
Figure 2. The arrangement that actually shipped.
Choosing the pattern from the content is harder than it looks
A pattern fixed in advance is a bet placed blind, so the natural improvement is to place it after seeing the hand. Divide the keys into contiguous blocks, summarize each cheaply — the mean of its keys is the usual choice — score a query against the summaries, and compute attention only inside the blocks that scored well. This is where recent work concentrates, and where the remaining headroom probably is.
It is also harder than the description implies, in three ways that compound. The selection costs something, and unless it is far cheaper than the attention it avoids the saving evaporates. A summary is lossy, so a block holding one highly relevant key among many irrelevant ones summarizes to nothing much and is skipped, silently. And a data-dependent pattern varies by query, head, and layer, so the work is ragged — awkward to batch, awkward to fuse.
That last point deserves more weight than it gets, because it decided the field. Sparsity a matrix multiply cannot exploit is not a saving. Accelerators earn their speed on dense contiguous tiles, and skipping scattered individual entries saves nothing: the tile is loaded either way and the multiply runs at the same rate regardless. Savings require block structure, coarse enough that a kernel can decline to load whole tiles. Hence every scheme still in use is block-shaped, and an elegant pattern with an excellent theoretical operation count is regularly slower than dense attention with a well-written kernel. Then the bar moved. While these methods were being developed, exact attention became several times faster and linear in memory by never materializing the grid , and an approximation must beat the best implementation of what it approximates, not the one it was benchmarked against.
None of these schemes restores what was given up. No arrangement of windows, strides, and hubs makes lookup of an arbitrary distant token reliable, so retrieval-like behaviour inside a long input degrades first — which is what long-context evaluations probe, and where sparse models look worst.
A pinch of math: the accounting behind a window
The comparison is elementary, and the numbers set the terms of the argument.
Dense attention performs $n^2$ score computations for a sequence of length $n$; a window of width $w$ performs $nw$. The ratio is
$$\frac{nw}{n^2} = \frac{w}{n}$$so the saving is real only when $n \gg w$. At $n = 2{,}000$ with $w = 512$ the factor is four, which is not worth an approximation. At $n = 200{,}000$ it is nearly four hundred, which is.
Now the reach. If each layer extends a position’s view by $w$, then after $\ell$ layers the receptive field is roughly $r \approx \ell w$, and requiring $r \ge n$ gives the depth needed to connect the two ends of the input:
$$\ell \ge \frac{n}{w}$$The consequence is a ceiling rather than a slowdown. Forty layers with a window of five hundred and twelve reach about twenty thousand tokens, and feeding the model two hundred thousand does not make the field grow: positions beyond it cannot influence each other at all, at any cost, however long the input is permitted to be.
Figure 3. The window saves nothing until the input is long.
What to carry forward
Sparse attention converts a sound observation into a gamble. The score grid pays for every pair of positions while the attention a trained model uses is overwhelmingly concentrated, so most of that arithmetic establishes irrelevance — but a weight is known to be negligible only because it was computed, so declining to compute a pair means deciding blind, before the input exists. The saving is guaranteed; the loss depends on the input and surfaces only when the discarded pair was the one that mattered. A sliding window makes cost linear and is defended by the receptive field growing with depth, as in a convolutional stack, but reach by relay is not reach by lookup: information must survive re-compression through intermediate positions, the bottleneck attention was built to remove. A pure window additionally collapses rather than degrades, since the earliest tokens serve as a sink for attention with nowhere to go, so streaming systems pin a few permanently beside the window. Strided and dilated patterns extend reach and lose resolution; global tokens give a long-range path that is narrow by construction; and the arrangement that survived into production is the least clever one, interleaving windowed layers with occasional full-attention layers, which divides the quadratic term rather than removing it. Content-dependent block selection is where the headroom lies and the difficulty concentrates: selection costs something, summaries hide single relevant keys, and a ragged pattern fights the fused kernels that make attention fast. The hardware objection is the binding one: sparsity without block structure saves nothing a matrix multiply can collect, and dense attention became several times faster while these schemes were being designed, raising the bar they had to clear. None of them recover reliable access to an arbitrary distant token, which is why retrieval inside a long input breaks first.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
