Articles

The memory accounting in the article on AdamW left something out. Weights, gradients, and optimizer state came to roughly sixteen bytes per parameter, which is a large number and a fixed one — it does not change with how much text you push through the model. Activations are the other half of the story, and they behave completely differently: they scale with the batch size and the sequence length, and for a large model on a long context they can dwarf everything else. The remedy is one of the more satisfying trades in the whole subject, because it exchanges a resource that is hard and expensive to obtain for one that is merely tedious. This piece is about why the backward pass needs so much stored material, and how throwing most of it away and computing it a second time turns out to be cheaper than keeping it. ...

Training a neural network is routinely described as finding the settings that minimize a loss, which makes it sound like a search — as though a map existed and the procedure consisted of reading it. There is no map. A model in training has access to exactly one thing: the value of the loss at the point it currently occupies, together with the local slope of that loss in every direction at once. It cannot see the next valley, or tell whether the ground ahead rises or falls. From that poverty of information the entire apparatus of training follows — the tiny steps, the schedules, the momentum, the cautious business of nudging a billion numbers a fraction of a percent at a time. What makes any of it affordable is a piece of arithmetic bookkeeping so efficient it is easy to mistake for a triviality. This piece is about the constraint that only local slope is available, the reverse sweep that recovers that slope for every weight at once, and what the resulting number withholds. ...

The usual account of attention’s cost is that it grows with the square of the sequence length, and the well-known responses are aimed at that square. A machine serving a large model to a few hundred people at once is not usually failing on arithmetic. It is failing on space. Somewhere in its memory sits one key vector and one value vector for every token, of every head, in every layer, for every conversation in flight, and that structure — not the parameter count, not the score grid — decides how many people the machine can hold and therefore what a token costs to sell. The remedy has to be storing less per token. The technique that delivered it rests on an observation close to an insult: most of what is stored is redundant. This piece is about which part of an attention head carries its specialization, what it costs to make heads share the rest, and why the saving turns out to be so nearly free. ...

Ask a model for the page number of a quotation it has never seen and it will give you one. Not a hedge, not a refusal, not a note that the request is beyond it — a page number, formatted correctly, inside a sentence of ordinary competent prose. The common reading is that the model is guessing badly, or was trained carelessly, or is in some loose sense pretending. None of those is quite right. The behaviour is what the training procedure asks for, at every position, and a model that did otherwise would have scored worse for it. That turns hallucination from a mysterious defect into a consequence of what the objective measures and what the available text contains. This piece is about why abstention is never the rewarded action, what the model’s own probabilities do and do not tell you about its uncertainty, and which of the available interventions actually change the situation. ...

Most claims about what happens inside a language model are inferences from behaviour: the model does this, so something in it must be doing that. One case is different. There is a mechanism whose components have been located, whose wiring has been traced, whose function has been stated precisely, and which can be switched off to confirm that the behaviour it explains disappears with it. It is not complicated — two attention heads in sequence, performing a copy — and it accounts for a surprising share of what makes these models useful, including the ability to follow a pattern demonstrated in a prompt. It also arrives during training in a way that is visible from outside, in a small kink in the loss curve that turns out to mark the moment a capability comes into being. This piece is about that mechanism and what its existence suggests. ...

QLoRA compressed a frozen model so that adapters could be trained beside it, and mixed precision worked out how few bits a model can be trained in. The third case is the one most people actually encounter: taking a model that is finished and will never be trained again, and making it smaller so that it fits on hardware someone owns. The situation is more forgiving than training, since there are no gradients to preserve and no accumulation to worry about — but it is also less forgiving in one specific way, because there is no opportunity to adapt. Whatever damage the compression does is permanent, and the model must live with it on every request. This piece is about how much can be taken away, why the activations turn out to be harder than the weights, and what the resulting formats actually buy. ...

Before the first gradient arrives, every weight in the model has to be set to something, and the something is drawn at random. This sounds like a detail to be got out of the way, and for a long time it was treated as one. It is not. The scale of those random draws determines whether a signal survives its journey up an eighty-block tower or dies on the way; it determines whether the gradient makes it back down; and — the part that turns a numerical nicety into an economic question — it determines whether a learning rate that worked on a small model still works on a large one. That last property has a name and a construction, and it is the reason nobody tunes hyperparameters on a frontier model directly. This piece is about how the starting point is chosen and why choosing it correctly lets you do your experiments somewhere cheap. ...

One quantity keeps turning up across the training of language models, wearing a different job each time. It is the imitation metric in distillation , the leash in DPO and in reinforcement learning from human feedback, one of the damage measures in ablation , and the term that training actually minimizes once cross-entropy is decomposed. Four unrelated-looking uses, one formula. It is worth understanding on its own terms rather than meeting it four times as a piece of notation, particularly because it has a property that surprises everyone the first time — it is not symmetric, and which way round you write it changes what you are asking for. This piece is about what the quantity measures, why the asymmetry is a feature rather than a defect, and how to tell which direction you want. ...

The generation loop takes one line to state: the model reads everything so far, produces a distribution over the vocabulary, one token is chosen and appended, and the whole thing runs again. Described that way it sounds tidy, and it hides something absurd. Taken literally, producing the thousandth word of an answer would mean pushing all nine hundred and ninety-nine earlier words back up through every block of the tower, from scratch, to compute representations the model had already computed nine hundred and ninety-nine times before. No system does this, and the thing that saves it is the KV cache — a running set of notes that turns a re-reading problem into a memory problem. This piece is about what is in those notes, why they consist of keys and values and nothing else, and why the memory problem they create now governs almost everything about what it costs to run a model. ...

The limit on human preference data is usually presented as a matter of money: annotation is expensive, budgets are finite, so the field went shopping for something cheaper. That framing gets the constraint wrong. What limits preference data is not its price but the fact that somebody has to read two long responses attentively enough to have a defensible opinion about which is better, and attention of that kind does not scale and cannot be bought in bulk. A second limit sits behind it: an annotator cannot demonstrate a good answer to a question they cannot answer, so on difficult material the ceiling is competence rather than diligence. The field’s answer was to hand the comparison to a model. This piece is about what that substitution leaves untouched, what it makes newly legible, and the moment its errors stop cancelling. ...