Articles

Short, self-contained essays on the techniques that make modern language models practical — each one intuition-first, with just a pinch of maths and a “what to carry forward” at the end. Read them in any order.

Supervised Fine-Tuning — Teaching a Model to Answer Rather Than Continue

Take a freshly pretrained model — one that has read a large fraction of the written internet, that can finish a legal clause or a Haskell function with unnerving fluency — and ask it a plain question. What often comes back is not an answer. It is another question, or three more questions, or a table of contents, or the same question rephrased and then a different one. The natural reading is that the model is being obtuse, or does not know, or has been loaded wrong. None of those is what happened. Documents containing a question very frequently continue with further questions, which is what a list of frequently asked questions is, and the model is doing exactly the thing it was trained to do: guess what comes next. It is not withholding an answer. It has never been told that answering is the activity. This piece is about the stage that installs that convention, why it needs so little data, and the ceiling it cannot pass. ...

Test-Time Compute — Buying Accuracy With Sampling

There is a familiar experience with a hard question. The model answers, the answer is wrong, you paste the identical prompt back in, and the second answer is right. Nothing changed. What happened is that a language model does not hold an answer, it holds a distribution over answers, and the reply on the screen is one draw from it. On a hard question it is spread across candidates, one of them correct, and the reply you were shown was the unlucky sample. Industrialized, that observation becomes the family of techniques filed under test-time compute: buying accuracy with extra forward passes when the question is asked rather than extra parameters when the model is built. This piece is about how the buying is done, what the price schedule looks like, and where the purchase stops working. ...

The Residual Stream — The Shared Channel Every Layer Reads and Writes

A transformer is usually described as a tower of blocks , each doing its work and handing the result to the next, and that picture is good enough for most purposes but quietly wrong in one respect. The blocks do not hand anything to each other. They all read from, and write to, a single vector that runs the full height of the model — entering at the bottom as an embedded token and leaving at the top to be turned into scores. Each block adds its contribution to that vector rather than replacing it. Once you see the architecture this way, as a shared channel with many contributors rather than a pipeline of transformations, a great deal that seemed arbitrary starts to look inevitable. This piece is about the residual stream: what it is, why addition rather than replacement turned out to matter so much, and what it explains about how these models are read and repaired. ...

The Transformer Block — The Unit That Gets Stacked

Descriptions of a language model tend to arrive as an inventory. There is attention, which moves information between positions; there is a feed-forward layer, which holds most of the parameters; there is normalization, which keeps the numbers in range; there is a residual connection, which lets the gradient travel. Each part gets explained on its own terms, and then a diagram appears showing eighty of something stacked in a column, and the column is called the tower. The step in between is skipped. Which is a shame, because that step is where the architecture is decided — the components are not arranged arbitrarily but in the one shape that makes a hundred copies of the same unit safe to pile on top of one another. This piece is about why the block alternates between mixing across positions and computing within them, what the two wrappers around those operations are for, and how much of the design is left unspecified. ...

Tokenization — Cutting Language Into Pieces a Model Can Count

Before a model sees anything, text has to be cut into pieces and each piece assigned a number. This is the only step in the entire pipeline that does not learn. The embedding table is trained, the attention weights are trained, the output map is trained; the tokenizer is a fixed recipe, built once from a sample of text by an algorithm with no notion of meaning, and thereafter frozen. Everything the model can express is expressed in units this recipe chose, and a surprising number of the odd behaviours people notice — trouble with arithmetic, trouble counting letters, uneven costs across languages, occasional total derangement on a particular string — trace back to it. This piece is about how the pieces are chosen and what the choice costs. ...

Tool Use — Letting a Model Call Something That Actually Knows

There are things a language model is bad at that a pocket calculator from 1975 does perfectly. Multiplying two ten-digit numbers. Telling you today’s date. Reporting what a particular file contains. These are not hard problems; they are problems of a kind the model is architecturally unsuited to, because it has no persistent state, no access to the world, and no mechanism for exact computation — only a very good sense of what text usually follows what. Tool use is the arrangement that lets it stop trying. The model writes down what it wants done, something else does it, and the result comes back as text the model can read. What makes this more than plumbing is that the interface between a probabilistic text generator and a piece of software that expects exact arguments is a genuinely awkward joint, and most of the engineering is about that joint. This piece is about how the call is made, how it is made reliably, and what the model still has to get right. ...