Articles

Short, self-contained essays on the techniques that make modern language models practical — each one intuition-first, with just a pinch of maths and a “what to carry forward” at the end. Read them in any order.

Learning-Rate Schedules — Deciding When to Slow Down

The learning rate is usually introduced as a number: one small constant, multiplied into every gradient and then left alone. Almost no large model has ever been trained that way. In practice it is a function of the step count — near zero for the opening few thousand steps, rising to a peak, then falling for the rest of the run — and the shape of that function is treated as somewhere between a solved problem and a superstition. It is neither. The opening ramp exists for a specific and slightly embarrassing reason to do with the optimizer’s own bookkeeping, and the conventional curve imposes a constraint so awkward that an entire alternative arrangement was invented to escape it. This piece is about why the rate has to move at all, what the two halves of the schedule are separately doing, and what the standard shape quietly costs. ...

Linear Attention — Trading Exactness for a Cost That Does Not Explode

The quadratic cost of attention has attracted more attempts at circumvention than any other single fact in this field, and most of them have failed. The idea is a piece of algebra rather than an approximation heuristic: attention involves multiplying three matrices, matrix multiplication is associative, and doing the multiplications in the other order turns a quadratic cost into a linear one. That much is exactly true and costs nothing. What costs something is the softmax sitting in the middle, which blocks the reassociation and must be removed or replaced — and the entire history of this line of work is the history of discovering what the softmax was doing that nobody had noticed. This piece is about the family that has come closest, about what the softmax turned out to be doing, and about why “closest” has been such a moving target. ...

Logits — The Scores Beneath the Odds

The shape of a forward pass is easy enough to state. A token comes in, the tower of blocks works on it, and out the far end comes a single arrow — a few thousand numbers holding everything the model has assembled about what should happen next. But a vector is not a word, and somewhere between that arrow and the token that finally appears on your screen there is a step that is easy to wave past. That step produces one number for every entry in the vocabulary, and those numbers are the logits. They are the last thing the model computes for itself, before any of it is turned into something a person would recognize as a prediction. This piece is about where they come from, what their name means, and the one property of them that trips up almost everyone who meets them. ...

Long Context — When the Window Outgrows the Attention

A model advertised as handling a million tokens is making two claims, and only one of them is usually true. The first is that it will accept an input that long without failing — a claim about position encodings, memory, and serving infrastructure, and one that is now routinely satisfied. The second is that it will use that input, attending to the relevant part of it as reliably at position nine hundred thousand as at position nine hundred. That claim is much weaker than the marketing implies, and the gap between the two is where most disappointment with long-context models lives. This piece is about what it takes to extend a window, what breaks when you do, and how to tell the difference between a model that can hold a document and one that can read it. ...

LoRA — Teaching an Old Model New Tricks Without Retraining It

A pretrained model arrives as billions of knobs, nudged downhill across a vast landscape until they settle into a capable configuration. It is rarely the configuration anyone wants. A model may need to answer in a house style, to reason about legal contracts, to write to a particular register, or to cover a speciality the general model knows only vaguely. The obvious fix is to keep training it on more specific data — a step called fine-tuning — and fine-tuning works. The trouble is that fine-tuning a giant model the ordinary way means adjusting all of its billions of knobs, which demands the same crushing hardware, memory and expense as the original training run, and produces a second full copy of an enormous model for every task. LoRA — short for Low-Rank Adaptation — sidesteps both costs by training a small set of new knobs and leaving the original untouched. This piece is about how that small set can be enough, what it saves, and what it cannot change. ...

Loss Spikes — When a Training Run Falls Over

The loss curves that appear in technical reports are smooth. They descend, they flatten, and they suggest that training a large language model is a matter of starting the machinery and waiting. The curves kept by the people running those jobs look different. Somewhere around step forty thousand, with nothing changed in the configuration, the loss jumps — not drifts, jumps — by a whole nat or two inside a handful of steps, and then does one of three things: claws its way back over a few thousand steps, settles permanently above the trajectory it left, or keeps rising until the model emits one token forever. This is a loss spike, and it is why a frontier run is watched on a rota rather than left alone over a weekend. This piece is about what happens when a run falls over, why the proposed causes are still argued about, and what the available defences cost. ...

Memorization — What a Model Keeps Verbatim, and When That Matters

Memorization arrived in machine learning with a settled meaning and a verdict attached: a model that had fitted its training set too closely, which was bad and, better still, detectable — watch the held-out loss, and when it turns upward while the training loss keeps falling, stop. That diagnostic is still taught, still correct on its own terms, and almost beside the point for a model trained once over a corpus it will never exhaust. The held-out loss does not turn upward. Nothing on the dashboard looks wrong. The model has nonetheless retained enough of its training data to reproduce parts of it word for word, including passages that appeared a handful of times and strings nobody wanted kept. The old alarm stays quiet because it was wired for a different fault. This piece is about why the training objective asks for verbatim retention rather than merely permitting it, what determines which sequences receive it, and what the available remedies do and do not remove. ...

Mixed Precision — Training in Half the Bits Without Losing the Thread

The article on QLoRA introduced the idea of describing a number in fewer bits, and applied it to a frozen model being adapted. Training a model from scratch in reduced precision is a harder problem, because a frozen weight only has to be read accurately while a weight under training has to accumulate millions of tiny corrections without any of them being lost to rounding. The tension is sharp: half-precision arithmetic runs several times faster and halves the memory traffic, which is the difference between a training run costing one month and three, but the updates the optimizer wants to apply are frequently smaller than the smallest change a half-precision number can represent. This piece is about how that tension is resolved, why the resolution hinges on a detail of how floating-point numbers are laid out, and what changed when the format itself was redesigned for the purpose. ...

Mixture of Experts — A Model That Only Switches On What It Needs

Almost every language model shares a quiet assumption: to process a single word, it uses all of itself. Every one of its billions of knobs fires for every word, whether that word is a comma or a rare medical term. This feels so natural it barely registers as a choice — but it carries a heavy price. The only way to make such a model know more is to make it bigger, and a bigger model is proportionally slower and costlier, because all of it runs, all the time, for everything. Mixture of Experts — usually shortened to MoE — is the family of models that breaks this bargain. Its idea is to build something vast in what it knows while only switching on a small slice of itself for any given word. This piece is about how that trick works, the surprisingly awkward problems it creates, and the tidy division of labour it does not deliver. ...

Multimodality — Teaching a Language Model to See

A transformer does not care what its inputs mean. It consumes a sequence of vectors and produces a sequence of vectors, and nothing in attention or the feed-forward layers makes any reference to language. The embedding table is the only component that assumes its inputs are tokens, and it is a lookup rather than a computation. Which means the path to a model that sees is narrower than it looks: if you can turn an image into a sequence of vectors that live in the same space as token embeddings, the rest of the machinery will process it without modification. The interesting question is not whether this works — it plainly does — but what it costs, what gets lost, and why a model that can describe a photograph in detail will confidently misread a clock face. This piece is about the join between a vision system and a language model, and what remains awkward about it. ...