Position is the detail most accounts of a transformer dispose of in half a sentence, somewhere near the embedding lookup, on the way to the interesting part. That treatment has it backwards. The mechanism at the centre of the architecture is not inattentive to word order but provably indifferent to it, making the dog bit the man and the man bit the dog one object as far as the arithmetic goes, and the fixes for that indifference are design decisions with measurable consequences — in how far a model can read, in how fast it runs, in what it quietly finds hard. A good deal of the difficulty with long inputs is a positional problem in other clothes. This piece is about why order has to be supplied from outside, what each scheme for supplying it has cost, and what none of them delivers.
Attention treats a sentence as a set
Attention works entirely by comparison. Every position emits a query and a key, each query is dotted against every key, and those scores become the weights deciding whose value gets pulled in. Read that description again and look for the part that mentions where a token sits. There is nothing. The score between two positions is a function of their two vectors and of nothing else, so shuffling the tokens of a sentence leaves the set of comparisons unchanged — the same pairs, the same scores, dealt into different slots. The property has a name. The mechanism is permutation-equivariant: permute the input and the output is permuted identically, with no other difference.
For a device meant to process unordered collections this is exactly right. For a model of language, where the difference between two orderings of five words is frequently the whole of the meaning, it is a startling place to begin.
The indifference was not an oversight, which is why it cannot simply be repaired. Permutation equivariance is the same property as parallelism: because no position’s computation depends on the order positions arrive in, all of them can be computed at once, which is what let the architecture be trained on hardware built for doing many identical things simultaneously, and what the recurrent designs it replaced could not do. Order was not left out of attention by accident. It was removed, deliberately and profitably, and then put back by other means. Every scheme below is a patch, and the patch is what parallelism costs.
The first schemes stamped position onto the input
The original transformer, from 2017, took the most direct route: if the mechanism cannot see position, write position into what it does see. Before the first layer, each token’s embedding has a position-dependent vector added to it, built from sines and cosines at geometrically spaced frequencies — some dimensions cycling every few positions, others taking thousands to come around once.
The appeal was considerable and largely theoretical. The encoding is a fixed function, so it adds no parameters and needs no training. It is defined for every integer, so nothing runs out. And it carries an elegant algebraic property: shifting position by a fixed amount is a fixed linear transformation of the encoding, so relative offsets are in principle recoverable.
The costs took longer to surface, and there are three. The signal is added to the vector that carries meaning, so position and content compete for the same dimensions of the residual stream . It is injected once, at the bottom, and must survive every layer intact enough to be usable at layer sixty. And the extrapolation the fixed functional form seemed to promise never arrived, because a well-defined encoding for an unseen position is not the same thing as a learned response to one.
Learned absolute encodings traded the last of these away entirely. Rather than compute the vector, store one trainable vector per position and let the optimizer decide what each seat means. In-distribution this works slightly better, since nothing forces position twelve to relate to position thirteen in any particular way. Outside the training length it fails absolutely rather than gradually: position four thousand and one has no row, and there is nothing sensible to put there. Not a degradation. A wall.
A pinch of math: what it means to depend on the gap
Write the attention score between a query at position $m$ and a key at position $n$ as $s(m,n)$. What every scheme here is reaching for is that the score should depend on the two positions only through their difference:
$$s(m, n) = h(x_m,\, x_n,\, m - n)$$The reason is linguistic rather than mathematical. The relations a model needs — the adjective before this noun, the subject at the head of this clause, the bracket that opened this one — are offsets, and they mean the same thing on page one as on page ninety. A scheme parameterized by absolute index encodes what the model does not want and relies on it to reconstruct what it does.
The sinusoidal scheme does not deliver that form, but it comes close enough to be instructive. Take one pair of dimensions driven at frequency $\omega_i$, so the encoding at position $m$ restricted to that pair is
$$p_m^{(i)} = \begin{pmatrix} \sin(m\omega_i) \\ \cos(m\omega_i) \end{pmatrix}$$Move forward $k$ positions and the angle-addition identities give
$$p_{m+k}^{(i)} = \begin{pmatrix} \cos(k\omega_i) & \sin(k\omega_i) \\ -\sin(k\omega_i) & \cos(k\omega_i) \end{pmatrix} p_m^{(i)}$$which is a rotation through $k\omega_i$ — a matrix depending on the offset $k$ and not at all on where we started. A shift in position is a fixed linear map on the encoding, the same map everywhere. The bare dot product of two encodings is cleaner still, collapsing to a sum of $\cos((m-n)\omega_i)$ across frequencies, a function of the gap alone.
Here is where property and practice part company. That dot product is not the attention score. The encoding was added to the content, so the vector entering the query projection is $x_m + p_m$, and the score expands into four terms: content against content, content against position, position against content, position against position. Only the last is a clean function of the gap, and even it arrives filtered through two learned matrices with no obligation to preserve the rotation. So the relative offset is linearly recoverable, and that is all. A map exists that would extract it; nothing compels the weights to become that map. The model must learn to use the offset from the lengths it happened to see, and the distance between an algebraic property and an exploited one is where the extrapolation went.
Figure 1. A shift in position is one rotation.
Moving the position inside the score
If the score is what should depend on the gap, the direct approach is to intervene on the score. Relative schemes do that: compute the content term as usual, then add a learned bias determined by the offset between the two positions, with distant offsets sorted into shared buckets so the count stays finite. This is the right idea, and was recognized as such early. It is also expensive in an awkward way.
The expense is structural rather than arithmetic. A relative bias enters inside the score computation — an offset-dependent term per pair of positions, per head, per layer, growing with the square of the length in exactly the place where nothing should. Worse, it requires indexing into the score matrix, the object the fused implementations under FlashAttention exist to avoid writing down. A positional scheme that obstructs the fastest available kernel is not merely slower. It is usually not adopted.
The simplest form survived on those grounds. Drop the learning and subtract a penalty proportional to the distance between the two positions, with a fixed slope per head, so some heads look narrowly and others far. It costs almost nothing, it is defined at any length, and models using it degrade gently past their training window. The catch is written into the mechanism: a monotone penalty on distance makes far-away tokens systematically harder to see, whatever they contain. As a prior over language that is broadly correct, since most dependencies are local. As a policy for finding one sentence buried in a hundred pages it is wrong by construction.
Rotary embeddings are the arrangement that won, and they win by turning the query and key vectors themselves, each by an angle proportional to its position, so that when the two are compared only the difference of the angles survives. The treatment is in the article on rotary embeddings . Why it displaced the others is the point here — relative behaviour from absolute bookkeeping, no parameters to store or exhaust, and an operation applied to the two vectors before any score exists, which leaves the fused kernels undisturbed.
Figure 2. Where each scheme injects position.
Figure 3. A monotone penalty is the wrong prior for one buried fact.
None of them extrapolates
The shared failure deserves bluntness, because the schemes are often ranked as though one had solved it. A model trained to four thousand positions does not work at forty thousand under any encoding. What it produces instead is attention that flattens or collapses, and text degrading in a way resembling confusion rather than arithmetic failure. The interventions that genuinely extend a window — rescaling the frequencies so a longer range maps onto angles the model has seen, or adjusting the base rate — are corrections applied with a little further training, as set out under long context . They are repairs, not properties the encoding possessed. The functional form guarantees a defined value at position forty thousand and one. It guarantees nothing about the response.
Two further findings sit underneath this, fixed by none of it. Models attend unevenly across position regardless of scheme, favouring the beginning and the end of an input over the middle, an asymmetry owing as much to the softmax normalizing over an ever-larger candidate set as to any encoding. And measured attention decays with distance faster than the encoding alone accounts for, which suggests the locality is partly learned from data in which the useful thing is usually nearby. The encoding supplies a coordinate system. What a model does with the far end of it is a separate question, largely empirical.
What to carry forward
Attention compares every query with every key on identical terms, nothing in that comparison refers to where a token sits, and shuffling a sentence leaves the set of comparisons untouched — the mechanism is permutation-equivariant, which for a model of language is a remarkable starting point and is also what makes every position computable at once. Order was removed to buy parallelism, then reinstated by other means, and each means charges for it. Fixed sinusoidal encodings cost no parameters but share the residual stream with meaning, must survive every layer after one injection at the bottom, and do not extrapolate despite an algebra suggesting they should. Learned per-position vectors do slightly better inside the training length and hit a hard wall beyond it, an unseen position having no entry and no substitute. Relative schemes encode the right quantity — the offset rather than the index — and pay for it inside the score computation, where cost grows with the square of the length and the fused kernels are obstructed; the cheap surviving variant, a fixed penalty on distance per head, buys graceful behaviour at new lengths by making distant tokens systematically harder to see, a sound prior for language and the wrong one for finding a fact in a long document. Rotary embeddings prevailed by delivering relative behaviour through absolute bookkeeping, adding no parameters, and acting on the query and key rather than the score, leaving the fast implementations available. The ranking is about cost and compatibility rather than capability. None of these schemes lets a model work at a length it never trained on; the extensions that appear to are rescalings applied with further training, and the uneven, distance-decaying way models attend across a long input is not something a coordinate system can fix.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
