Descriptions of a language model tend to arrive as an inventory. There is attention, which moves information between positions; there is a feed-forward layer, which holds most of the parameters; there is normalization, which keeps the numbers in range; there is a residual connection, which lets the gradient travel. Each part gets explained on its own terms, and then a diagram appears showing eighty of something stacked in a column, and the column is called the tower. The step in between is skipped. Which is a shame, because that step is where the architecture is decided — the components are not arranged arbitrarily but in the one shape that makes a hundred copies of the same unit safe to pile on top of one another. This piece is about why the block alternates between mixing across positions and computing within them, what the two wrappers around those operations are for, and how much of the design is left unspecified.
Depth is where the capability comes from, and naive depth destroys the signal
The reason to want many layers rather than one enormous one is composition. A single layer, however wide, applies one round of processing; it can recognize patterns but it cannot build a pattern out of the result of recognizing a pattern. Behaviour that takes several dependent steps — identify the subject, then decide which sense of the verb applies — needs those steps in sequence, each able to see the output of the last. Depth buys sequence. Past a certain point capability tracks depth, not width.
A naive stack does not survive being deep. If every layer replaces its input with its output, two things go wrong, both worse with height.
Information has to be copied to be kept. Early layers establish what later layers still need — what the token literally is, which of its senses is in play. Under pure replacement every layer must reproduce all of that or lose it, so a large part of its capacity goes on preservation rather than computation. Preservation becomes a job, relearned at every layer.
The gradient becomes a long product. Training requires the error signal at the top to reach the weights at the bottom, and through a chain of pure transformations that journey is a product of many terms. Products of many terms are badly behaved: slightly below one and the product vanishes, slightly above one and it explodes. Either way the lowest layers receive noise. For years this was the ceiling, and the reason was arithmetic rather than any shortage of ideas.
The block’s shape is not a matter of taste. It is a response to two failures the architecture did not choose.
Mix, then think, then mix again
The computational content of a block is two operations, and what makes them a pair is that each does exactly what the other cannot.
Attention moves information between positions and does no private processing. Every position takes a weighted average of information drawn from the others, with content-dependent weights. The operation is still a mixture: it decides where to look and copies what it finds. It has almost no capacity to transform what it collected, since the only transformation available is linear, and a linear map composed with a weighted average is another weighted average.
The feed-forward layer processes each position independently and sees nothing else. Two matrix multiplications with a nonlinearity between them, applied to one token’s vector as though that token were alone. No interaction between positions whatsoever. The nonlinearity is what lets it recognize a pattern in the vector it was handed and add something learned in response — which is where a model’s factual knowledge is found when people go looking.
Mix, then think. Then mix again, on the results of everyone else’s thinking. Take either half away and what remains is not a weaker model but a different and much poorer machine.
Attention with no feed-forward layer is a weighted-averaging device with nowhere to put knowledge. Stack it as often as you like and every output is still a blend of the input embeddings shuffled through linear maps. It can route beautifully and has nothing to route — no place to keep that Paris is in France. Strip the feed-forward layers out of a trained model and performance does not degrade gracefully; it falls off a cliff.
A feed-forward layer with no attention is a per-token lookup that cannot see context. Every token would be processed identically regardless of what surrounded it, which reduces the model to an elaborate dictionary — everything about the word bank in general, nothing about which sense is meant here. Contextual meaning is the entire problem.
The alternation is what lets each half cover the other’s blind spot.
Figure 1. Each half covers the other’s blind spot.
Two wrappers make the stacking survivable
Around each of those operations sits the same pair of accessories, and neither computes anything.
The residual connection makes addition the default. Each sub-layer adds its output to what came in rather than returning it alone. A block’s default behaviour is therefore to pass everything through unchanged: a sub-layer that computes nothing useful does no damage, so preservation is free and interference takes effort. The gradient benefits in the same stroke, since differentiating an addition yields the identity plus a correction rather than a multiplication — a hundred blocks compose additively instead of multiplicatively, and the long product becomes a sum with a direct path to every layer.
Normalization keeps the scale in range. Every block adds and nothing subtracts, so the magnitude of the accumulated vector grows as it rises, sometimes by an order of magnitude between bottom and top. A sub-layer fitted against inputs of one typical size behaves differently when handed one ten times larger, so a rescaling step puts the input into a known state first.
What mattered more than the operation was where it sits. The original arrangement normalized the sum, which puts a rescaling directly on the residual path and fits the express route with a gate at every floor; such models needed long warm-ups and could diverge outright. Moving the normalization inside the branch — rescaling only what feeds a sub-layer, leaving the addition untouched — trains stably at depths where the older placement will not train at all. The cost follows from the same property: nothing on the path rescales, so the magnitude keeps growing and each successive block contributes proportionally less, until the upper layers of a very deep model are barely felt.
A pinch of math: two residual updates and where the parameters sit
The whole block, written out, is two lines of the same form. Let $x$ be the vector arriving for one position:
$$x \leftarrow x + \text{Attn}(\text{Norm}(x))$$$$x \leftarrow x + \text{FFN}(\text{Norm}(x))$$Every symbol is doing a specific job. The $\text{Norm}$ supplies each sub-layer with an input of predictable scale, and it sits inside the parentheses so that the addition stays clean. The $x +$ makes the identity the default and the gradient’s path additive. $\text{Attn}$ is the only term that reads other positions; $\text{FFN}$ is the only term with the nonlinearity needed to compute something new. The arrow is assignment: the same vector is updated twice, so a block reads better as two small edits to a running total than as one transformation.
The parameter count lands somewhere counter-intuitive. Write $d$ for the model’s width. Attention needs four projections — query, key, value, output — each of size $d \times d$:
$$4d^2$$The feed-forward layer expands fourfold and comes back, so its two matrices are $d \times 4d$ and $4d \times d$:
$$8d^2$$Twelve $d^2$ per block, two thirds of it feed-forward and one third attention. That ratio survives the usual modifications: gated variants add a third matrix and shrink the hidden width to compensate, grouped-query schemes shave the key and value projections to reduce cache size, and neither moves the split much. Attention receives the overwhelming majority of the explanation and holds the minority of the weights.
Two costs fall out of the same accounting. Compute per token is roughly twice the parameter count, about $24d^2$ per block, identical for every token in every context. And attention adds a term with no parameters at all but a cost proportional to $n^2 d$ for a sequence of length $n$ — quadratic, invisible in the weight count, and the reason long contexts are expensive in a way the parameter budget never reveals.
Figure 2. Two residual edits to a running total.
Figure 3. Where the parameters actually sit.
What the block cannot decide, and what it will not do
Having a well-shaped unit does not tell you how to use it, and the gaps are specific.
Nothing says how many to stack, or how to divide a budget between depth and width. The block is scale-agnostic by construction; the same two lines work at width one thousand and at width sixteen thousand. Which combination to build is not derivable from the architecture and is settled empirically, by fitting curves to the losses of many trained models — scaling laws are the record of that fitting, not a consequence of the design. Nothing in it dictates how many heads to divide the width among either, or how positions are told apart, which is why rotary schemes drop in without disturbing its shape.
Every token costs the same. A block applies its full complement of arithmetic to the word the and to the final digit of a long multiplication with equal diligence. There is no mechanism for a token to skip a block, no way to spend more computation on a hard position than an easy one. Early-exit and conditional-depth proposals exist; none has become standard, partly because the batching that makes inference and parallelism efficient assumes every position takes the same path. Mixture-of-experts varies which parameters a token consults, which is a different lever — it does not let a token consult fewer blocks.
There is no recurrence and no state. A block holds nothing between forward passes. Whatever a model appears to remember it is re-deriving from the tokens in front of it, every time, and the cache that makes generation tractable is an arithmetic saving rather than a memory. Nor can a block write anywhere but the running vector, a fixed-width channel all hundred instances compete for.
What to carry forward
The block has the shape it has because depth is the only route to composed, multi-step behaviour and a naive deep stack destroys the signal twice over — every layer must spend capacity copying what it preserves, and the gradient’s descent becomes a long product that vanishes or explodes. Its computational content is an alternation, and the alternation is the design: attention mixes information across positions and can barely transform what it collects, while the feed-forward layer processes each position in isolation and cannot see any other. Neither is sufficient. Attention alone is a weighted-averaging machine with nowhere to keep knowledge; feed-forward alone is a per-token lookup blind to context. Two wrappers make the stacking survivable: the residual connection, which has each sub-layer add rather than replace, so a block’s default behaviour is to pass everything through and blocks compose additively; and normalization, whose placement — inside the branch rather than on the residual path — mattered considerably more than which formula was used. The parameter split runs two thirds feed-forward to one third attention, the reverse of where the explanation usually goes. The costs are structural. Compute per token is fixed regardless of how hard the token is, with no way to skip a block. Attention’s quadratic term is invisible in the weight count and dominant at long context. Nothing in the design specifies how many blocks to stack or how to split a budget between depth and width, which is settled by scaling laws fitted after the fact. And the block carries no state between forward passes: whatever the model seems to remember, it is rebuilding from scratch, every time, from the tokens it can see.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
