A transformer is usually described as a tower of blocks , each doing its work and handing the result to the next, and that picture is good enough for most purposes but quietly wrong in one respect. The blocks do not hand anything to each other. They all read from, and write to, a single vector that runs the full height of the model — entering at the bottom as an embedded token and leaving at the top to be turned into scores. Each block adds its contribution to that vector rather than replacing it. Once you see the architecture this way, as a shared channel with many contributors rather than a pipeline of transformations, a great deal that seemed arbitrary starts to look inevitable. This piece is about the residual stream: what it is, why addition rather than replacement turned out to matter so much, and what it explains about how these models are read and repaired.
The problem with a tower that overwrites
Suppose each block genuinely did replace its input with its output. Then everything the model knows at layer thirty would have had to pass through twenty-nine successive rewritings, each free to discard whatever it did not find useful.
The first cost is informational. Early layers do a lot of work that later layers still need — identifying what the token literally is, resolving its part of speech, noticing which of its senses is in play. If layer four’s output must contain both its own contribution and a faithful copy of everything from layers one to three, then a large part of every block’s capacity is spent on copying rather than computing. Preservation becomes a job, and a job that must be relearned by every block.
The second cost is worse and is the one that historically stopped people from building deep networks at all. To train a stack of transformations, the gradient has to travel from the loss at the top all the way back to the bottom, passing through every block on the way. In a chain of pure transformations that journey is a long product of terms, and a long product of terms is a numerically hostile object: if the terms are typically a little below one the product vanishes toward zero, and if a little above one it explodes. Either way the layers at the bottom receive a signal that is useless. Networks past a couple of dozen layers were, for years, simply not trainable, and the reason was this.
Addition instead of replacement
The fix is one line of arithmetic. Have each block compute its contribution and then add it to what came in, rather than returning it alone. Write $x$ for what arrives and $f$ for the block:
$$x_{\text{out}} = x_{\text{in}} + f(x_{\text{in}})$$The default behaviour is now to pass everything through unchanged. A block that computes nothing useful contributes nothing and does no damage; preservation is free and interference is the thing that requires effort. Nobody has to learn to copy.
The effect on the gradient is just as direct. Differentiating that line gives the identity plus the derivative of $f$ — so the gradient reaching a block’s input is the gradient at its output plus a correction, rather than the gradient at its output multiplied by something. Instead of a long product that vanishes or explodes, there is a direct path from the loss to every layer, with each block’s contribution added along the way. This is why the bottom of a hundred-layer network receives a usable signal at all, and it is the single change that made depth practical. The blocks that follow the residual connection through the tower are, from the gradient’s point of view, an express route with local exits.
Figure 1. A tower that overwrites, and one that adds.
The stream as a shared channel
Once you accept that every block adds to the same vector, the natural way to think about that vector changes. It is not a representation being progressively refined by a pipeline. It is a communication channel, or a shared workspace, that every component in the model can read from and write to.
The picture is worth spelling out. The stream begins as a token’s embedding. An attention head reads some of what is there, computes something, and adds a vector back in. A feed-forward layer reads the updated stream, computes something else, adds that. Sixty or a hundred such contributions accumulate, and at the top the unembedding reads whatever the total came to. The stream at any given layer is a sum of everything written so far, and — this is the part with real consequences — that sum can often be decomposed back into its parts, because addition is reversible in a way that a general transformation is not.
Two things follow immediately.
Components can communicate across long distances in the tower without the layers in between being involved. A head in layer three can write something that a head in layer twenty reads, and the seventeen layers between them neither need to carry it deliberately nor can they easily prevent it. This is how a model builds multi-step behaviour: not by a chain of blocks each handing to the next, but by earlier components leaving material in the channel for later ones to pick up. Circuits in the interpretability sense are exactly this — sets of components, often far apart, that have arranged to write and read the same thing.
And the stream has a fixed width, which makes it a scarce resource. Every component is writing into the same few thousand dimensions. There is no partitioning, no allocation table, nothing preventing two components from writing to overlapping directions and interfering with each other. That the model manages to store far more distinguishable pieces of information than it has dimensions is a genuine puzzle, and it has a name and an answer of its own.
Figure 2. The stream as a channel, not a pipeline.
A pinch of math: reading and writing with subspaces
How does a component write something that another can retrieve without everything else in the channel getting in the way?
The answer is geometry. When a block computes its contribution, the last thing it does is multiply by an output matrix, and the columns of that matrix determine which directions in the stream the contribution can lie along. A component effectively writes into a subspace. Symmetrically, when a component reads, it multiplies the stream by an input matrix whose rows determine which directions it is sensitive to.
Suppose a component writes a vector $\alpha v$ into the stream, and a downstream component reads by projecting onto a direction $u$. What the reader receives from that write is:
$$u^\top (\,x + \alpha v\,) \;=\; u^\top x \;+\; \alpha\, (u^\top v)$$The message gets through in proportion to $u^\top v$ — how aligned the writing direction is with the reading direction. If the two are aligned, the reader sees the message clearly. If they are perpendicular, the reader sees nothing at all: $u^\top v = 0$, and the write is invisible to this particular reader while remaining perfectly legible to any reader aligned with $v$.
That orthogonality is the whole trick. Two components can share the same channel without interfering, provided they write along directions that are perpendicular to what the other’s readers are looking at. In a space of four thousand dimensions there is a great deal of room to be mutually perpendicular, and the model, under nothing but the pressure to reduce loss, arranges its writers and readers accordingly. The $u^\top x$ term — everything else currently in the stream — is noise from this reader’s point of view, and it is small precisely because most of what is in the stream lies in directions this reader does not care about.
Figure 3. Perpendicular writers do not disturb each other’s readers.
What this explains
Several things that look like separate facts stop being separate once the stream is in view.
The logit lens works because the stream is in the same coordinates all the way up. Applying the unembedding to a partly-accumulated stream is meaningful because what you are reading is a genuine partial sum — the model’s prediction as of layer twelve, which is exactly what the top of the tower would see if the remaining blocks all contributed nothing. The prediction is drafted early and amended because amendment is literally what the architecture does.
Ablation and activation patching work for the same reason. Zeroing a head’s contribution removes one term from a sum, leaving the others intact, and the difference in output is attributable to that term. In a stack of pure transformations there would be no such clean subtraction to make; the contribution would be entangled with everything downstream of it.
And the practice of thinking about directions rather than neurons — probing for a direction that encodes sentiment, or steering a model by adding a vector to its stream mid-forward-pass — is only sensible because the stream is a linear space that components address by direction. Adding a vector to the residual stream is not a hack from outside the system. It is precisely what every block in the model does.
Two costs
The picture is not free of trouble, and two problems are worth naming.
The first is contention. Fixed width and many writers means the channel can become crowded, and there is evidence that models actively manage this — components that appear to do nothing but clear away material that is no longer needed, and heads whose apparent function is to suppress what another head wrote. Some of the strange behaviour found in trained models is bandwidth management rather than computation.
The second is scale. Because every block adds and nothing subtracts by default, the magnitude of the stream tends to grow as it rises through the tower, sometimes by an order of magnitude or more between bottom and top. Left alone this destabilizes everything downstream, since a block trained to read a stream of one typical size behaves differently when handed one ten times larger. Keeping the stream in a workable range at every level is a job in its own right, and it is why a normalization step sits at the entrance to every block — reading a rescaled copy of the stream while the stream itself continues, un-rescaled, up the tower.
What to carry forward
The blocks of a transformer do not form a pipeline handing results to one another; they all read from and write to a single vector running the height of the model, the residual stream, and each block adds its contribution rather than replacing what was there. That one choice makes preservation the default and interference the thing requiring effort, and it turns the gradient’s journey from a long product that vanishes or explodes into a sum with a direct route to every layer — which is what made deep networks trainable in the first place. Read as a shared communication channel, the stream explains how distant components coordinate: an early head writes something a much later head reads, with the layers in between neither carrying it deliberately nor able to stop it, and that is what a circuit is. Components address the channel by direction, writing into the subspace their output matrix spans and reading the subspace their input matrix selects, so two components can share the same fixed-width channel without interfering as long as their directions are close to perpendicular — which is also why so much of interpretability is phrased in terms of directions rather than neurons. The same linearity is what licenses reading the stream partway up with the unembedding, and what makes ablating one component a clean subtraction from a sum. The costs are a crowded channel that models appear to manage actively, and a magnitude that grows steadily with depth, which is the reason a normalization step guards the entrance to every block.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
