Articles

Every block in a transformer is preceded by a small operation that rescales its input, and it is the least glamorous thing in the architecture. It computes no features, stores no knowledge, and would be omitted from any diagram drawn on a napkin. It is also the difference between a model that trains and one that does not. The residual stream grows as it rises, because every block adds and nothing subtracts; a block trained to read a signal of one typical size will misbehave when handed one ten times larger; and the gradient inherits every one of those distortions on the way back down. This piece is about the operation that keeps the numbers in range, the variant that discarded half of it and lost nothing, and the placement decision that mattered more than either. ...

A frontier model does not fit on one machine, and the arithmetic makes the point without ceremony: the article on AdamW ’s sixteen bytes per parameter puts a four-hundred-billion-parameter model at six and a half terabytes of training state, against perhaps eighty gigabytes on a single accelerator. Something has to be divided. What makes this more than a logistics problem is that there are five fundamentally different things you can divide — the data, the width of each matrix, the depth of the tower, the sequence itself, and, where the model has them, the experts — and each division has a completely different communication bill attached. Getting the combination right is worth a factor of several in training time, and getting it wrong produces a cluster of expensive machines mostly waiting for each other. This piece is about the five axes and how they are stacked. ...

Position is the detail most accounts of a transformer dispose of in half a sentence, somewhere near the embedding lookup, on the way to the interesting part. That treatment has it backwards. The mechanism at the centre of the architecture is not inattentive to word order but provably indifferent to it, making the dog bit the man and the man bit the dog one object as far as the arithmetic goes, and the fixes for that indifference are design decisions with measurable consequences — in how far a model can read, in how fast it runs, in what it quietly finds hard. A good deal of the difficulty with long inputs is a positional problem in other clothes. This piece is about why order has to be supplied from outside, what each scheme for supplying it has cost, and what none of them delivers. ...

Two of the better-known methods in post-training define themselves against this one: DPO exists to avoid it, and RLVR is it with the reward model replaced by a checker. The thing itself is worth describing directly. Proximal policy optimization arrived from robotics, where the problem was teaching a simulated body to walk without the training process destroying itself, and it turned out to transfer to language with remarkably little modification. The core difficulty it solves is the same in both settings and is worth stating before any of the machinery: in reinforcement learning, unlike in supervised learning, the model generates its own training data, so a bad update does not merely produce one bad step — it changes what the model will see next, and a policy that has degraded will collect worse experience, from which it will degrade further. Supervised learning is forgiving because the dataset does not move. This piece is about how to take the largest step you safely can when it does. ...

Ask why language models are trained to predict the next word and the answers tend toward the grand: that language is produced sequentially, that prediction is compression and compression is understanding, that anticipating what comes next is what a mind does. None of that is exactly wrong, and none of it is the reason. The objective won a competition against several serious rivals, on grounds that were largely accounting — how much learning can be squeezed out of each token of text, and out of each unit of compute. For a couple of years around 2018 the favourite was something else entirely. Why the sums came out as they did explains a good deal that otherwise looks arbitrary: why encoder models still beat decoders at certain jobs, why a model is never trained in the situation it is deployed in, and why a handful of losing objectives survive in particular corners. This piece is about the constraint that forced the choice, the arithmetic that settled it, and what the field gave up in exchange. ...

LoRA removed most of the price of adapting a giant model: only a small pair of narrow grids trains, and the original stays frozen. That collapses the training bookkeeping to almost nothing. One obstacle survives it. Training a small set of new knobs still requires holding the entire frozen model in memory throughout — all those billions of original numbers have to sit there so the calculations can flow through them. For the largest models that alone can demand hundreds of gigabytes, far beyond a single ordinary graphics card. QLoRA — Quantized Low-Rank Adaptation — is the trick that knocks down this last wall. The idea is direct: if the frozen model is what consumes the memory, store the frozen model in a more compact form. This piece is about how the compression works, why the rounding survives it, and what the saving does not reach. ...

A model’s knowledge is fixed at the end of training, distributed across billions of weights in a form nobody can inspect, edit, or attribute. It cannot tell you where it learned something. It cannot be told that a fact has changed without retraining. And when it does not know an answer it will frequently produce one anyway, because nothing in its training taught it to distinguish confident recall from confident invention. Retrieval-augmented generation addresses all of this by an intervention that is almost embarrassingly direct: before asking the model to answer, go and find the relevant documents, and put them in the prompt. Nearly all the difficulty, however, lives in the retrieval step and in the seam between the two systems, and the naive version fails in ways that are entirely predictable once that seam is understood. This piece is about where the pipeline actually breaks, what the seam demands of both halves, and when retrieval is the wrong instrument to reach for. ...

An ordinary language model has a curious habit: it answers instantly. Give it a question and it produces the next word, then the next, marching straight from the prompt to a reply with no visible pause for thought. For a great many tasks this is fine — recalling a fact, rephrasing a sentence, drafting an email. But ask an ordinary model to multiply two large numbers, or to work through a tricky logic puzzle, or to debug a subtle piece of code, and it often stumbles, blurting out a confident answer that happens to be wrong. It behaves like a student who has been forbidden from using scratch paper and must announce the answer the instant the question is read aloud. Reasoning models are what appears when that ban is lifted. They are trained to think first — to spend time working a problem through, step by step, before committing to an answer. This piece is about what “thinking” means for a machine that only knows how to predict words, how such models are made, and what the extra room does not buy. ...

Anyone who learned this subject from the textbooks of the twenty-tens learned that the central hazard of fitting a flexible model to data was overfitting, and learned an apparatus for holding it off: a penalty on the size of the weights, noise injected into the activations, a held-out split watched for the moment its curve turns up, augmentation to manufacture examples the collection lacked. That training was sound and the techniques worked. Then you read a technical account of a large language model and find almost none of it. No noise injection. No stopping criterion based on held-out loss, because nothing turns upward to stop at. A weight decay coefficient, retained, listed among the settings with no explanation beyond its value. The tempting conclusion — that the received wisdom was wrong, or that scale makes carelessness affordable — is not what happened. This piece is about which precondition stopped holding, what the surviving weight decay is actually doing, and what took over the job the old defences performed. ...

The article on embeddings concerned the table at the bottom of a language model, which turns each token into a vector so the tower has something to work with. The object here is a different one with an unfortunately identical name: a model whose entire output is a single vector representing a whole passage, built so that passages about the same thing land near each other and can be found by searching. The two share a word and a geometric intuition and almost nothing else. The distinction is worth insisting on, because a great deal of confusion in practice comes from expecting one to do the other’s job — and in particular from the reasonable-seeming and wrong idea that a good language model must contain a good search index. This piece is about why it does not, what the training objective actually asks for, and what the resulting vectors cannot be made to do. ...