Most claims about what happens inside a language model are inferences from behaviour: the model does this, so something in it must be doing that. One case is different. There is a mechanism whose components have been located, whose wiring has been traced, whose function has been stated precisely, and which can be switched off to confirm that the behaviour it explains disappears with it. It is not complicated — two attention heads in sequence, performing a copy — and it accounts for a surprising share of what makes these models useful, including the ability to follow a pattern demonstrated in a prompt. It also arrives during training in a way that is visible from outside, in a small kink in the loss curve that turns out to mark the moment a capability comes into being. This piece is about that mechanism and what its existence suggests.
The behaviour to be explained
Give a model a sequence in which something has already happened once, and it will predict that it happens the same way again. Show it ... Dr. Aisha Rahman ... Dr. Aisha and it will produce Rahman. Show it a list where every entry follows a template and it will continue the template. Give it three examples of a made-up transformation and it will apply the transformation to a fourth.
The general rule is: find where this current token occurred earlier in the context, and predict whatever followed it then. Stated so plainly it sounds trivial, and it is not, for two reasons. The name being copied may be one the model has never seen — invented for the occasion — so this cannot be retrieval from training. And it is the mechanical core of in-context learning: a model that can find a previous instance of the current situation and repeat what came next is a model that can be taught by demonstration inside a prompt, without any weights changing.
Why it takes two heads
A single attention head cannot do this, and understanding why is the crux.
Suppose the current token is Aisha and we want to find the earlier Aisha and copy what came after it. A head attending directly to the earlier Aisha retrieves that token’s value — which tells us about Aisha, which we already know. We do not want the matching token. We want its successor.
The head could try attending to Rahman directly, but on what basis? Rahman is not the token we are holding and there is nothing about it to match on. The information required — that Rahman is preceded by Aisha — is not present at Rahman’s position. It has to be put there.
So the mechanism is built in two stages, in different layers.
A previous-token head, in an early layer, does something almost aggressively simple: at every position, attend to the position immediately before and copy information about it into the residual stream
. After this layer, the position holding Rahman also carries a record that its predecessor was Aisha. Every position now knows what came before it.
An induction head, in a later layer, does the actual work. Its query, formed from the current token Aisha, is matched not against other tokens’ identities but against the previous-token record that the earlier head deposited. So it asks: which position is preceded by Aisha? That question matches at Rahman’s position. The head attends there, copies Rahman’s value forward, and the unembedding
turns it into a prediction of Rahman.
The two heads form a circuit: distinct components in different layers, communicating through the residual stream, jointly implementing a function that neither performs alone. The first head writes into a subspace; the second reads that same subspace. This is precisely the mechanism described in the article on the residual stream — components coordinating across depth by agreeing on a direction — and it is the clearest worked example of it.
Figure 1. The two-head circuit that copies a successor.
A pinch of math: matching on a shifted key
The composition can be made precise, and it is worth doing because it shows the trick is one substitution rather than anything elaborate.
An ordinary head computes attention scores between position $i$ and position $j$ as
$$s_{ij} = (W_Q x_i) \cdot (W_K x_j)$$where $x$ is the residual stream at that position. If the stream mostly carries token identity, this matches token against token — useful for many things, useless here.
The previous-token head has modified the stream. At position $j$ it has added a term encoding the identity of token $j-1$, along some direction the head writes into. Write the stream as approximately $x_j \approx e_j + \tilde{e}_{j-1}$, where $e$ denotes the token’s own contribution and $\tilde{e}$ the shifted copy in its own subspace.
Now the induction head’s key matrix $W_K$ is trained to read the shifted subspace and ignore the direct one, while its query matrix $W_Q$ reads the direct one. The score becomes
$$s_{ij} \;\approx\; (W_Q e_i) \cdot (W_K \tilde{e}_{j-1})$$which is large when $e_i$ and $e_{j-1}$ describe the same token. The head attends to position $j$ when position $j-1$ matched the current token — exactly the required behaviour.
The remaining piece is the copying. The head’s value and output matrices are arranged so that whatever token is at the attended position is written back into the stream in a direction that promotes that same token in the output. Composing the two matrices gives an operator on token space whose eigenvalues skew positive: read a token, promote that token. This can be inspected directly in a trained model, and the fraction of the eigenvalue mass that is positive serves as a quantitative signature distinguishing real copying heads from heads that merely look similar. Note the condition is only that the spectrum leans positive — not that the operator approximates the identity, which would be far stronger. In large models with feed-forward layers this weight-based analysis becomes unwieldy, and heads are identified behaviourally instead, by a copying score and a prefix-matching score measured on activations.
The bump in the loss curve
The most striking thing about these heads is that you can watch them form.
Early in training, the loss falls smoothly. Then, over a narrow window of a few hundred steps, there is a small bump — a brief plateau or even a slight rise, followed by a faster drop. It is easy to miss and appears reliably across model sizes, and it had no accepted explanation for some time.
It is the induction heads forming. Before the window, the model has no way to use repeated structure in its context. During it, the two-stage circuit assembles. After it, the model can copy from context, and its loss drops because a great deal of text contains repetition that was previously unexploitable.
Two things make this more than a curiosity. First, in-context learning ability — measured as how much better the model predicts late tokens in a sequence than early ones, which is precisely the benefit of having context to learn from — appears at the same moment. The capability and the mechanism arrive together. Second, ablating the induction heads removes most of that ability — though this is where the account is weaker than it is usually reported to be. The ablations were run on small models, and are strongest for attention-only ones; no ablation was performed at production scale, and for large models the evidence is correlational and mechanistic rather than causal. The authors themselves note the co-occurrence could reflect a shared cause, such as the model learning to compose layers through the residual stream at that moment. Taken at small scale it is still about as complete a story as interpretability has produced about anything: a behaviour, a mechanism, a moment of formation, and an intervention that removes both together.
It also gives a concrete answer to a question that is otherwise vague. Capabilities that appear to emerge suddenly with scale or training may correspond to specific circuits assembling at specific moments. Not all of them will — and the article on emergent abilities sets out the good deflationary argument that much apparent emergence is a measurement artifact — but here is one case where a discontinuity in behaviour really does correspond to a discrete thing coming into existence.
Figure 2. A bump in the loss, and a capability arriving.
What generalizes, and what does not
The simple copy is the beginning rather than the end. In larger models the same machinery is found doing more abstract versions of the same operation: matching not on exact token identity but on similarity, so the pattern completes for a paraphrase rather than a repeat; matching on position within a repeated structure, so a template is followed rather than a string; operating over several tokens rather than one. The family is broad and the boundaries are not crisp.
Related circuits have been traced by the same methods. The most thoroughly documented is the one behind indirect object identification — deciding, in when Mary and John went to the shop, John handed a drink to ___, that the answer is Mary — which involves a chain of head types including some that identify names, some that suppress duplicates, and the name-mover heads that do the final copying. That is the circuit reconstructed in the worked example of the article on ablation
, and it was found with the same combination of ablation and patching.
The honest limit is scale. These circuits were mapped in small models, on narrow tasks, by people spending months on each. Nobody has a full account of a production model, and the reason is superposition : components in a large model are polysemantic, participating in many circuits at once, so isolating one is far harder than in a two-layer model built for study. The programme of finding circuits by hand does not obviously scale, which is why the field has shifted toward automated decomposition — sparse autoencoders to find the features first, and automated search for the connections between them. Induction heads remain the proof that there is something there to find.
What to carry forward
An induction head implements a rule of the form: find where the current token appeared earlier in the context, and predict whatever followed it then. This works on tokens the model has never seen, so it is not retrieval from training, and it is the mechanical core of learning from examples inside a prompt. It requires two heads because a head attending to the earlier occurrence retrieves that token rather than its successor, and the fact that a given position is preceded by a particular token is not available at that position until something puts it there — so a previous-token head in an early layer copies each position’s predecessor into the residual stream , and an induction head in a later layer forms its query from the current token and matches it against that shifted record, attending to whatever position was preceded by a match and copying it forward. The two communicate through a shared subspace, which is the clearest worked example of a circuit. The composed value-and-output operator of a real induction head has eigenvalues that skew positive — read a token, promote that token — which serves as a quantitative signature in small models, though at scale heads are identified behaviourally instead. The heads form abruptly during training, producing a visible bump in the loss curve, and in-context learning ability appears in the same window and disappears when they are ablated in small models — a causal account at that scale, and a correlational one at production scale, where the ablations were never run. Larger models contain more abstract versions matching on similarity or structure rather than identity, and related circuits have been mapped by the same methods — but all of it has been done in small models on narrow tasks, and the polysemanticity of large ones is why the field has turned toward automated decomposition instead.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
