Appendix E — Compute-Optimal Scaling: The Chinchilla Story
Chapter 9 touched on the question of how to split a training budget between a bigger model and more data, and named the Chinchilla correction. This appendix tells that story in full, because it is one of the most illuminating episodes in the field — a case where the conventional wisdom was confidently wrong, got corrected by a single careful study, and then turned out to have a twist that the correction itself missed.
The surprise that started it: quality you can forecast
One of the strangest and most useful discoveries in modern AI is that the quality of a language model does not improve in fits and starts as you make it bigger and train it longer. It improves smoothly — and, crucially, predictably. Run a handful of small, cheap experiments, plot how good the models get as you spend more, and the line is so regular that you can extend it forward and estimate, with real confidence, how good a model ten or a hundred times larger will be, before you have spent a penny building it.
This is a remarkable thing to be able to do. In most of engineering you cannot test a paper airplane and confidently predict a jumbo jet. But in this corner of AI the small experiments really do tell you about the big ones. That predictability is what turned model-building from guesswork into planning. The central question stops being “will this work?” and becomes “given what I am willing to spend, what is the smartest way to spend it?”
Figure 1. Run small experiments, extend the line, forecast the big model.
The fixed-budget choice: a bigger brain or more reading
Here is the trade-off at the heart of everything. Building a model costs a fixed pot of compute — think of it as a fixed number of hours on expensive machines. With that pot you face one unavoidable choice: spend it on a bigger model (more internal knobs, more raw capacity) or on more training data (showing the model far more text). You cannot maximize both — every unit that buys a bigger brain is a unit that cannot buy more reading.
The cleanest analogy is a student with a fixed number of study-hours before an exam. They can grow a bigger memory — more raw mental capacity — or spend those same hours reading more material. A huge memory stuffed with very little reading is wasted potential; a tiny memory drowning in more books than it can absorb is also wasted. Somewhere in between is the sweet spot, and the whole game is finding it.
Figure 2. One fixed pot, three ways to split it.
The early advice: mostly just make it bigger
When researchers first worked this out, around 2020, their answer was lopsided: when you get more compute, pour most of it into making the model bigger, and only a little into extra data. Build a giant brain; do not worry too much about over-feeding it.
The field took this to heart, and the result was a generation of famous models that were enormous but, in hindsight, badly underfed — giant brains that had simply not read enough. The best-known example was a model of around a hundred and seventy-five billion knobs trained on a relatively thin diet of text — only a couple of words of training data per knob. Picture a student with an extraordinary memory who skimmed only a couple of textbooks: impressive raw capacity, badly underused. These models were genuinely good and genuinely groundbreaking, but they were leaving performance on the table, and nobody quite realized how much.
The Chinchilla correction: grow them together
Then came the study that reset the field’s thinking, nicknamed after its model, Chinchilla. The researchers re-ran the whole experiment more carefully, and fixed a subtle flaw: in the earlier work, the data-heavy experiments had been quietly handicapped by a technical detail in how training was scheduled, which made “more data” look less valuable than it really was. Once that was corrected, the picture changed completely.
The new conclusion was a model of balance: model size and training data should grow roughly in step with each other. Don’t lavish everything on the brain and starve it of reading — scale both together. This crystallized into the single most quoted number in the field: roughly twenty words of training data for every knob in the model. It is a back-of-the-napkin rule, and that is exactly why people love it — it turns a hard planning problem into a one-line estimate.
The team then delivered the knockout. They built a model four times smaller than a rival flagship but, using the same compute budget, spent the savings on feeding it far more text — and the smaller, better-read model beat the larger, underfed one across the board. Back to the student: a kid with a more modest memory who actually did the reading outscored the prodigy who had barely cracked a book, and both spent the same hours studying. That single result was the proof, and after it “balance size and data” became the field’s default instinct.
The twist nobody saw coming: training-optimal is not serving-optimal
Here the story takes its most important real-world turn, because everything above is about getting the best model for a fixed training budget. It says nothing about what happens after the model is built, when it is actually being used — and that distinction turns out to be everything.
A trained model gets run to answer questions, write text, and power products — potentially billions of times. And every one of those runs costs more for a bigger model. A bigger brain is not just more expensive to build; it is more expensive to operate, forever, with every query it ever answers. So if you know a model will be used heavily and for a long time, the math flips. It becomes worth deliberately building a smaller model and then over-training it — feeding it far more than the tidy twenty-words-per-knob rule suggests, well past the point that is efficient for training. You knowingly “waste” training money to end up with a small model that is permanently cheap to run, yet still genuinely smart because it read so much.
This is the philosophy behind the famous Llama-style models: small brains trained on staggering amounts of text, sometimes a hundred or more words per knob, far beyond what the balanced rule recommends. It looks wasteful on training day, but amortized across billions of future uses it is the bargain of the century. The student analogy, in its final form: if you are cramming for one exam tomorrow, balance is best; but if you will be answering questions from that same knowledge every day for the rest of your life, it is worth pouring in extra reading now to keep a lean, fast, cheap-to-run mind forever. The right answer depends entirely on whether the big cost is the studying or the lifetime of using what you learned. So the balanced rule is not wrong — it is just answering a narrower question than people assume.
Figure 3. The balanced rule answers a narrower question than people assume.
The sobering footnote: running low on books
One quiet assumption underpins this entire story: that there is an endless supply of fresh, high-quality text to read. The whole “just feed it more” logic works only if there is more genuinely good material to feed. But that is starting to bend. The supply of high-quality human-written text is finite, and the most ambitious models are beginning to bump against its edges. Re-reading the same material does not help much — and past a point it actively hurts, the way cramming the same chapter for the tenth time stops teaching you anything. So the strategy that powers today’s best models — small, lean, and enormously well-read — leans on a resource the world is beginning to run short of. The reading list, it turns out, is not infinite.
The one-line takeaway
We learned to forecast a model’s quality before building it; learned the core trade-off is brain-size versus reading; corrected an early bias toward giant, underfed models with the balanced “twenty words per knob” rule; then realized that for models we will run forever it is smarter to build them small and over-feed them — all while quietly approaching the limits of how much good text there is left to read.
← Appendix D · Where Words Sit · Contents · Glossary →
The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.
