Every block in a transformer has two halves. Attention gets nearly all the discussion, and the other half — two matrix multiplications with a nonlinearity between them, applied to each token independently — gets described as “the feed-forward part” and passed over. This is an odd allocation of attention, given that the feed-forward layers hold roughly two thirds of the model’s parameters, and given that when researchers go looking for where a model stores the fact that Paris is in France, this is where they find it. Attention moves information between positions. The feed-forward layers are where information is looked up, transformed, and decided upon. This piece is about the half of the architecture that does the knowing.
What it is
The operation is the simplest thing in the model. Take the token’s vector, multiply by a matrix that expands it to a larger width, apply a nonlinearity, multiply by a matrix that projects it back down:
$$\text{FFN}(x) = W_{\text{down}} \; \sigma\!\left( W_{\text{up}} \, x \right)$$The expansion is conventionally fourfold — a model of width four thousand expands to sixteen thousand and comes back. Two matrices of four thousand by sixteen thousand is about 134 million parameters per block, against roughly 67 million for the attention projections of the same block. Multiply by eighty blocks and the ratio holds: two thirds of the model, give or take, is this.
The crucial structural fact is that this operates on one token at a time. There is no interaction between positions here at all; the same function is applied independently to every token in the sequence, exactly as it would be to a single token in isolation. Whatever a token has gathered from its neighbours, it gathered during the attention step. The feed-forward layer then works on the result privately.
That division gives the block a rhythm worth naming. Attention is the part where tokens consult one another; the feed-forward layer is the part where each token, alone, does something with what it heard. Communication, then reflection, repeated eighty times.
Figure 1. Two thirds of the parameters are not attention.
Why the nonlinearity is the whole point
Remove $\sigma$ and the layer collapses. Two matrix multiplications in a row are a single matrix multiplication — $W_{\text{down}} W_{\text{up}}$ is just some other matrix — so without the nonlinearity, the expansion and contraction buy nothing whatsoever, and indeed the entire tower would collapse to one linear map no matter how many blocks it had. Everything a network can do beyond drawing a straight line comes from the nonlinear function sitting between its linear ones.
Which function has drifted over the years, and the drift is instructive. The original transformer used ReLU, which zeroes anything negative and passes anything positive unchanged — brutally simple, and it works. GELU smoothed the corner, weighting each input by roughly the probability that a standard normal falls below it, which gives a small negative dip just below zero and a differentiable transition. The smoothing helps in practice, though the theoretical case for it has always been thinner than its adoption suggests.
The more interesting change was gating. Instead of one expansion, compute two in parallel, pass one through a nonlinearity, and multiply the two together elementwise:
$$\text{SwiGLU}(x) = W_{\text{down}}\Big( \sigma(W_{\text{gate}}\, x) \odot (W_{\text{up}}\, x) \Big)$$One branch produces content, the other produces a multiplicative gate deciding how much of that content passes. This is more expressive than a fixed nonlinearity because the gate depends on the input: the layer can learn to suppress a feature in one context and admit it in another, rather than applying the same fixed curve everywhere. It costs a third matrix, so implementations shrink the hidden width — typically to about two-thirds of what it would otherwise be — to keep the parameter count level. Nearly every recent model uses a gated variant, and the empirical gain is consistent, if modest. The original paper introducing them concluded, memorably, that it offered no explanation for why they work beyond divine benevolence, and the situation has not improved much since.
Figure 2. Gating adds a second branch and a narrower hidden width.
A pinch of math: the layer as a key-value store
Here is the reading that makes the feed-forward layer legible, and it comes from writing the two matrices in terms of their rows and columns rather than as blocks.
Let $k_i$ be the $i$-th row of $W_{\text{up}}$ and $v_i$ the $i$-th column of $W_{\text{down}}$. Then the whole operation can be rewritten as a sum:
$$\text{FFN}(x) = \sum_{i=1}^{d_{\text{ff}}} \sigma\!\left( k_i \cdot x \right) \, v_i$$Read that carefully, because it is the same shape as attention. Each hidden unit $i$ compares the input against a stored direction $k_i$ by dot product; the nonlinearity turns that comparison into a coefficient; and that coefficient scales a stored output vector $v_i$ which gets added to the total. The layer is a lookup table with sixteen thousand entries, where $k_i$ is what the entry matches on and $v_i$ is what it contributes when matched.
The difference from attention is where the keys and values come from. In attention they are computed from other tokens in the sequence, so the lookup is over the present context. Here they are learned weights, fixed after training, so the lookup is over everything the model absorbed from its corpus. Attention retrieves from what is in front of you; the feed-forward layer retrieves from memory.
This is not merely a suggestive analogy. Probing the hidden units of trained models finds that individual units respond to interpretable patterns in the input — a unit that fires on text about a particular topic, one that fires at the end of a quoted phrase, one that fires on a specific syntactic construction — and that the corresponding output column, read through the unembedding , promotes a coherent set of tokens. The units really do behave like rules of the form when the input looks like this, push the prediction toward that.
Figure 3. Attention retrieves from context, FFN from memory.
Editing a fact
The strongest evidence for the memory reading is that you can act on it.
If a specific fact is stored as a particular association between a key direction and a value vector in a particular layer, then locating and rewriting that association should change the fact and leave everything else alone. This turns out to be broadly possible. Using the causal tracing methods described under ablation , one can find which layer’s feed-forward output is carrying the answer at the moment the subject of a factual statement is being processed, then solve directly for a modification to $W_{\text{down}}$ that maps the relevant key to a different value. The model then reports the new fact, in varied phrasings, while its behaviour on unrelated inputs is largely unchanged.
Two qualifications keep this honest. The edits are less clean than the demonstrations suggest — they often fail to propagate to logical consequences of the edited fact, and applying many in sequence degrades the model. And the localization is real but partial: facts are distributed across several layers and several units rather than sitting in one place, so “the layer where this fact lives” is an approximation to a smeared reality. Still, the fact that targeted edits work at all is difficult to explain under any account except that these layers are storing associations.
Why so wide, and what the width is for
The fourfold expansion looks arbitrary and has survived a great deal of scrutiny, which suggests it is near something real. The reason it needs to be wide is that the number of distinct patterns a model wants to recognize vastly exceeds the width of the residual stream it has to read from. Four thousand dimensions cannot hold four thousand independent concepts if the model needs a hundred thousand of them, and the hidden layer is where the stream gets temporarily unpacked into a much larger space where more of those patterns can be separated before being compressed back down.
That “more patterns than dimensions” pressure is a subject in its own right, and the fact that the model resolves it by packing features into near-perpendicular directions rather than dedicated neurons is the reason individual hidden units are usually only partly interpretable. A unit that fires on three unrelated things is not malfunctioning; it is one axis of a space in which the model has stored more directions than it has axes.
The mixture-of-experts architecture is best understood as an intervention on exactly this layer. Its “experts” are feed-forward layers — the attention is left alone — and the observation motivating it is that for any given token, only a small fraction of those sixteen thousand hidden units fire meaningfully. If most of the lookup table is irrelevant to most queries, there is no reason to consult all of it. Splitting the layer into many smaller tables and routing each token to a couple of them keeps the total stored knowledge while consulting only a slice, which is precisely the “cheap to compute, expensive to store” bargain described in that article. That the sparse-model literature is entirely about this half of the block, and barely touches attention, is a good indication of where the parameters and the knowledge actually are.
What to carry forward
The feed-forward half of each block — two matrices with a nonlinearity between them, applied to each token independently — holds roughly two thirds of a model’s parameters and is where its learned knowledge sits. Its position in the block gives the architecture its rhythm: attention is where tokens exchange information, and this is where each token, alone, works on what it received. The nonlinearity is not a detail but the entire reason the layer exists, since two consecutive linear maps are one linear map; the field has moved from ReLU through GELU to gated variants that compute a content branch and a multiplicative gate in parallel, which are consistently a little better for reasons nobody has satisfactorily explained. Written out row by row, the layer is a sum over hidden units in which each unit compares the input against a stored key direction and, in proportion to the match, adds a stored value vector — the same shape as attention, except that the keys and values are learned weights rather than other tokens, so the lookup is over what the model absorbed in training rather than over the present context. That reading is supported by probing, which finds units responding to interpretable patterns and promoting coherent sets of tokens, and by the partial success of targeted factual edits that rewrite one key-value association in one layer. The width exists because a model needs to distinguish far more patterns than the residual stream has dimensions, and the observation that only a small fraction of hidden units matter for any given token is exactly what mixture-of-experts exploits — which is why sparse architectures modify this half of the block and leave attention untouched.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
