Articles

Almost all the attention paid to language models concerns the machine: how it is shaped, how it is optimized, how it is served. What goes into it is the part with the least published detail and quite possibly the largest effect. Two organizations can train the same architecture at the same scale with the same optimizer and produce models of visibly different quality, and the difference is the corpus. The decisions involved are not glamorous — what fraction should be code, how many times may a document be repeated, what to do with the internet’s vast supply of near-duplicate junk, what to feed the model in its final week — and they are made with far less theory than any other part of the pipeline. This piece is about what those decisions are and the surprisingly small amount that is actually known about them. ...

A model never produces a word. What comes out of the tower is a set of scores, one per vocabulary entry, and after the softmax a set of odds — a forecast, not a decision. Something outside the model has to read that forecast, commit to a single token, and then do it again, and again, until the text is finished. That something is the decoding strategy, and it is governed by a handful of dials that appear in almost every interface and API you will meet. This piece is about what each of them does, why several of them exist to solve the same problem in different ways, and why not one of them makes a model smarter. ...

Nearly every language model in use writes the same way: one token, then the next, each conditioned on everything before it and on nothing after. That constraint is so fundamental it is easy to mistake for a fact about language rather than a choice about architecture. It is a choice, and it has costs — a model cannot revise a word it has already committed to, cannot plan the end of a sentence before writing its beginning, and must spend one full forward pass per token no matter how obvious the token is. Image generation abandoned sequential production years ago in favour of starting with noise and refining the whole canvas repeatedly. This piece is about what happens when the same idea is applied to text: why the translation is harder than it looks, what the working version actually does, and why the results are interesting without yet being decisive. ...

The machinery that makes a large model good is well enough known: attention letting every word survey every other, a tall tower of blocks, and a long, expensive training run that settles billions of knobs into a capable configuration. Capability comes at a price. The best models are enormous, and every answer burns memory and time in proportion to their size — tolerable in a demonstration, punitive across billions of requests. So the wish is obvious: keep most of what a giant model knows while paying for a small one. Distillation is the technique that grants it, by transferring the large model’s judgement into a smaller one. This piece is about how the transfer works, why it works better than the size difference suggests, and what the student inherits along with the skill. ...

There is a boundary running through post-training. Where an answer can be checked by a program, verifiable rewards give a training signal that cannot be flattered, and we can push on it as hard as we like. But most of what we ask a model to do has no checker. There is no program that decides whether an explanation was pitched correctly, whether a refusal was gracious, whether a summary kept the right things. For that territory — which is most of the territory — we are back to human preference, and the standard way of learning from it is a three-stage apparatus with a reinforcement learning loop at the end. Direct preference optimization is the discovery that the whole apparatus collapses into a single loss you can train the way you train anything else. This piece is about how something that looks like reinforcement learning turns out, on inspection, to be classification. ...

A model does not see words. Long before the first block of the tower does anything, each token has been replaced by a list of a few thousand numbers pulled from a table with one row per entry in the vocabulary. That table is the embedding, and it is the model’s only point of contact with language as we write it. What is easy to miss is that nobody wrote the table — it begins as noise and is learned by exactly the same downhill walk that trains everything else, with one peculiarity in how the gradient reaches it. What is easier still to miss is that most models keep two such tables rather than one. This piece is about how a word finds its place, and about why reading a word and reaching for a word turn out to be different skills. ...

For a few years the standard illustration of what scale buys you was a plot in which a line lies flat along the bottom of the chart for three or four orders of magnitude and then, past some particular model size, turns upward and climbs almost vertically. The claim attached to it was that the ability being measured is simply absent below that point and present above it — that something switched on. The plot did an enormous amount of work. It appeared in talks about safety, in arguments about regulation, and in forecasts of what the next training run would do. It is also, for a large fraction of the abilities it was drawn for, a property of how the vertical axis was defined rather than of the models along the horizontal one. That does not settle the matter, because a residue survives the objection and the residue is the interesting part. This piece is about what the original claim asserted, the deflationary account that dismantles most of it, and what is left standing. ...

Almost everything else in the machinery of a language model can be verified. A gradient is correct or it is not; a cache holds the right keys or it does not; a quantized model produces outputs close to the original or it does not. Evaluation is the one activity where the ground keeps moving. We build models to be generally capable, then measure them with tests that are necessarily specific; the tests leak into the training data; the numbers rise while the thing being measured may not have improved; and the community moves to a new test and repeats the cycle. None of this is fraud, and most of it is not even carelessness — it is the predictable consequence of optimizing against a measurement. This piece is about why evaluation is genuinely hard, which failures are structural, and what can actually be trusted. ...

Every block in a transformer has two halves. Attention gets nearly all the discussion, and the other half — two matrix multiplications with a nonlinearity between them, applied to each token independently — gets described as “the feed-forward part” and passed over. This is an odd allocation of attention, given that the feed-forward layers hold roughly two thirds of the model’s parameters, and given that when researchers go looking for where a model stores the fact that Paris is in France, this is where they find it. Attention moves information between positions. The feed-forward layers are where information is looked up, transformed, and decided upon. This piece is about the half of the architecture that does the knowing. ...

Most improvements to attention have been approximations: restrict which positions may see which, or replace the mechanism with something cheaper that behaves roughly the same. This one is not. It computes the identical result to ordinary attention , to the last bit, and it is several times faster and uses a fraction of the memory. That combination should be suspicious — you do not usually get exactness and speed together — and the resolution is that the original implementation was not slow because of the arithmetic it performed. It was slow because of the data it moved. This piece is about the difference between those two things, and about the reordering of the computation that makes the largest intermediate result never need to exist. ...