Articles

There is an old and reliable way to find out what something does: remove it and see what breaks. Surgeons and neuroscientists have done this for centuries — lesion a small region of tissue, watch which ability fades, and infer that the region was responsible for it. The word is ablation, literally a carrying-away, and it has become one of the few tools that gets inside a large language model at all. A modern model is a black box of billions of interacting knobs, and nothing can be read off it directly. So the next best thing gets done instead. A piece is taken away — a layer, an attention head, a training trick, a single direction in the model’s internal space — the model runs again, and the damage is measured. The size of the damage is the measure of the part’s importance. This piece is about how that removal is defined, how the number is read, and how little a small number proves. ...

Training a neural network is a downhill walk. The knobs start at random, the loss says how wrong the current settings are, and backpropagation hands us, for every knob in the tower, a small number saying which way to nudge it. What it never says is how far. Real training does not take that number at face value; it accumulates momentum and keeps a separate step size for every knob, and that machinery has a name, AdamW, which is the optimizer behind very nearly every large model trained in the last several years. This piece is about the two ideas it combines, and about the single-letter correction — the W — that turned a good optimizer into the default one. ...

A language model, in its ordinary use, receives an input and produces an output. An agent is what you get when the output is fed back in — when the model acts, observes what happened, and decides what to do next, repeatedly, until it judges the task finished. The change sounds procedural and is not. A single response either succeeds or fails and the user sees which; a loop of thirty steps can fail at step four and spend twenty-six more steps building on the failure, and nobody notices until the end. This piece is about that loop, about the arithmetic of compounding reliability that governs how long it can run, and about the fact that most of the difficulty turns out to be context management rather than reasoning. ...

The cost of running a model is almost always quoted in operations. Training budgets are priced in floating-point operations, architectures are compared by operations per token, and halving the operation count of a component sounds unambiguously like progress. A great deal of work built on that premise produced no speedup, and some of it produced a slowdown. The arithmetic units on a current accelerator stopped being the scarce resource some time ago; they spend most of their existence waiting for numbers to arrive from memory, far more slowly than they can be consumed. Once that gap is wide enough the cost of a computation is set by how much data it touches rather than how much work it does with it, and a cost model counting only the work will misidentify what is expensive with near-perfect reliability. This piece is about the ratio that decides which of the two limits binds, why the same model sits on opposite sides of it in training and in generation, and what that fact explains about everything built on top. ...

Attention is the mechanism everything else in a transformer is arranged around, and it is almost always invoked as though it were already understood — named, granted, and moved past. It repays being paid for properly. The mechanism is not complicated — it is three linear projections, a dot product, a softmax, and a weighted sum, and you could write it on the back of a napkin. What is remarkable is how much follows from it: that a model can resolve a pronoun forty words back, that it can follow an instruction given at the top of a page, that the whole architecture parallelizes well enough to be worth training on a trillion tokens. This piece is about the mechanism itself, the reasoning that produced it, and the two things it conspicuously cannot do on its own. ...

The gradient the model actually receives is never the gradient of the thing we want to minimize. What we want is the average loss over all text; what we compute is the average loss over a few hundred thousand tokens drawn at random. The second is an estimate of the first, and how many tokens go into it is a choice — the batch size. Larger batches give a cleaner estimate and fewer steps per unit of data; smaller batches give a noisier estimate and more steps. It sounds like a straightforward accuracy-versus-cost trade, and it is not, because the noise turns out to be useful, because the returns from reducing it stop abruptly at a threshold that can be calculated, and because on the hardware these models run on the choice is entangled with how many machines you are using. This piece is about that threshold and what sits on either side of it. ...

A reasoning model shows its working, and the working is the most reassuring thing about it. Ask something hard and there it is on the page: an approach considered, a difficulty noticed, a first attempt discarded, a conclusion reached. It reads like an account of how the answer was arrived at, and it is hard not to read it that way: the format is borrowed wholesale from how people explain themselves, and nobody has practice at reading fluent prose as anything other than a report. Then someone runs the obvious experiment. Slip a hint into the prompt, watch the answer move to match it, and read the accompanying reasoning: a clean, sensible argument that never mentions the hint. Nothing malfunctioned. The model did what it was trained to do. This piece is about the difference between reasoning that does work and reasoning that describes work, why the second was never trained for, and what that leaves of the case for reading traces at all. ...

A team builds an agent that works beautifully for six steps and then begins repeating itself, and the diagnosis they reach for is that the model is not clever enough. They try a larger one. The repetition continues, because the problem was never the reasoning: by step twelve the original instruction sits forty thousand tokens back, sharing a window with three stale directory listings and the full text of a file the agent read once for a reason it can no longer recover. The name that has settled on the business of managing this is context engineering, which sounds like prompt writing with a grander vocabulary and is not quite that. What distinguishes it is that the decisions have measurable costs and trade against one another, which is what makes something an engineering problem rather than a set of habits. This piece is about the budget every component competes for, the ordering that the cache and the attention pattern argue about separately, and the losses that compaction and tool output quietly impose. ...

Accounts of what makes a language model fast tend to stop at the model. Cheaper attention kernels, fewer bits per weight, a small model guessing ahead — all of it aimed at making one forward pass cost less. Then they note, almost in passing, that serving is affordable because requests are batched, as though a batch were something you assemble and hand over. It is not. Requests arrive at times nobody chose and stop after wildly different numbers of tokens; one wants a yes or a no, the next a thousand-word summary. Nearly every difficulty in production serving descends from that variance, and the two mechanisms that deal with it are a scheduler and a memory allocator rather than anything inside the network. This piece is about how a batch is kept full when its members keep leaving, and about the allocator that decides how many members it may have. ...

Every number in a model is where it is because of one quantity. The loss is what training minimizes, what scaling laws predict, what a training run is monitored by, and what all the machinery of optimization exists to descend. For a language model that quantity is cross-entropy, and it is usually introduced as a formula to be accepted rather than a thing to be understood. That is a shame, because it has an exact and rather beautiful interpretation — it is the number of bits you would waste encoding real text using the model’s beliefs about text — and once that reading is in hand, several otherwise mysterious facts about training curves and model evaluation become obvious. This piece is about what the loss measures, why it and not something simpler, and what its familiar cousin perplexity adds. ...