Almost all the attention paid to language models concerns the machine: how it is shaped, how it is optimized, how it is served. What goes into it is the part with the least published detail and quite possibly the largest effect. Two organizations can train the same architecture at the same scale with the same optimizer and produce models of visibly different quality, and the difference is the corpus. The decisions involved are not glamorous — what fraction should be code, how many times may a document be repeated, what to do with the internet’s vast supply of near-duplicate junk, what to feed the model in its final week — and they are made with far less theory than any other part of the pipeline. This piece is about what those decisions are and the surprisingly small amount that is actually known about them.
Filtering: most of the internet is not worth reading
The raw material is a web crawl, and a web crawl is mostly not prose. It is navigation menus, cookie banners, boilerplate footers, machine-generated listings, keyword spam, and text so degraded that no human would read it voluntarily. A common crawl snapshot reduces, after processing, to somewhere between one and ten percent of its original volume. The pipeline that performs the reduction is a sequence of increasingly expensive filters.
The cheap ones come first. Strip the markup. Detect the language and keep what you want. Apply heuristics that catch the obvious: documents too short to contain an idea, documents whose lines mostly lack terminal punctuation, documents with an implausible ratio of symbols to words, documents consisting largely of a repeated line. These rules are inelegant and remove an enormous amount of material for very little compute.
Then deduplication, which matters more than its dullness suggests. The web repeats itself relentlessly — syndicated articles, mirrored documentation, quoted passages, templated pages differing in one field. Exact duplicates are easy. Near-duplicates require approximate matching, typically by hashing overlapping word sequences in a way that makes similar documents collide, then discarding all but one member of each cluster. Deduplication improves models substantially at fixed compute, and it does so through two mechanisms worth separating: it stops the model spending capacity memorizing text it has seen forty times, and it prevents contamination of held-out evaluations by material that also appears in training.
Last comes quality classification, which is where judgement enters. The usual approach trains a small classifier to distinguish crawl text from some reference corpus assumed to be good — reference-quality encyclopedic articles, well-regarded link aggregators, curated book collections — and keeps the documents it scores highly. It works, and it is worth being honest about what it does. A classifier trained to recognize text resembling a particular reference is a machine for encoding whose writing counts as good, and the resulting corpus inherits every bias in that choice. More recently the classifier is often a language model asked to rate educational value directly, which is more flexible and no less a judgement call.
Figure 1. A crawl reduced to a few percent.
Mixing: the proportions are the model
Having assembled sources — filtered web text, code, books, scientific papers, reference works, mathematics, multilingual material, and now large volumes of synthetic text — you must decide how much of each the model reads. These proportions are among the most consequential numbers in the whole enterprise and among the least published.
Two findings are robust enough to state. The first is that code helps things that are not code. Models trained with a substantial fraction of source code perform better on reasoning tasks with no programming content. The explanation usually offered is that code is unusually well-structured text with long-range dependencies, explicit compositional structure, and a strong correspondence between form and consequence — a good curriculum for whatever underlies multi-step inference. The explanation is plausible and not proven; the effect is measured repeatedly.
The second is that quantity and quality trade against each other at a rate that depends on how much compute you have. Aggressive filtering leaves a smaller, better corpus. If the budget is small relative to the data available, filter hard, since you will not exhaust even the strict subset. If the budget is large enough that you would have to repeat the strict subset several times, the calculation changes and a looser filter that supplies fresh tokens may win. This interacts directly with the scaling arithmetic, and it is why the right filtering threshold is not a property of the filter but of the run.
Attempts to set the proportions systematically exist. One approach trains a family of small models on different mixtures and fits a predictive relationship between mixture and loss, then optimizes it. Another treats the mixture as adjustable during training, reweighting toward sources where loss is still falling fastest. Both are real improvements over guessing and neither is decisive, and the proportions in a frontier model remain substantially a matter of accumulated institutional judgement.
Repetition: how many times may a token be read
If the corpus is smaller than the training budget, documents must be repeated. How much repetition is tolerable turns out to have a reasonably clean answer.
Up to about four passes over the same data, the loss achieved is close to what an equivalent volume of fresh data would have given — a penalty of well under one percent. The repeated tokens are nearly as valuable as new ones. Returns then decay, with meaningful gains persisting to roughly sixteen passes and collapsing sharply after; by forty passes additional epochs contribute essentially nothing, and runs pushed that far can begin to degrade outright.
This has a direct consequence for planning, and it also frames the much-discussed question of running out of text. The supply of high-quality human-written material is finite and models are consuming it at a rate that has been extrapolated to exhaustion within a few years. The four-epoch result means the wall is further off than a naive count suggests, since the effective supply is several times the nominal one. But it is a multiplier, not a reprieve.
Figure 2. Four passes are nearly free, forty are worthless.
Synthetic data, and the thing to be careful about
The response has been to generate text rather than find it, and it is now a large fraction of what goes into a model.
The useful cases are specific. Rewriting existing documents into cleaner or more instructive form. Generating problems and solutions in domains where correctness can be checked, which connects directly to verifiable rewards — a solution that passes its tests is a solution worth training on, and the checker means the data can be trusted without a human reading it. Producing coverage for situations that are rare in natural text. Distilling a stronger model’s outputs, which is distillation by another route.
The hazard is generating text with a model and training the next model on it without any external check. Each generation is sampled from the last, and sampling from a distribution and refitting to the samples loses the tails — rare constructions are drawn rarely, so they are under-represented in the sample, so the next model assigns them even less probability, so they are drawn less often still. Repeat this and the distribution narrows toward its own mode. The effect has been demonstrated cleanly in controlled settings, and the conditions under which it appears are the important part: it requires replacing real data with synthetic and closing the loop. Accumulating synthetic data alongside real data, or filtering it through any external criterion, largely prevents it. Which is why the practice that works pairs generation with verification, and the practice that fails is generation alone.
Ordering: the model’s last week matters most
The final decision is not what to include but when, and it is the one that has changed most recently.
Data order matters because a model trained by gradient descent is influenced most by what it saw last, especially as the learning rate decays toward the end of a run and each remaining step commits the weights more permanently. This has been turned into a deliberate technique. Rather than sampling uniformly from a fixed mixture throughout, training now commonly divides into a long bulk phase on the broad, cheap, heavily-filtered corpus, followed by a shorter annealing or midtraining phase in which the mixture shifts sharply toward the highest-quality material — textbooks, curated mathematics, verified code, instruction-like text — while the learning rate decays to near zero.
The gains from this final phase are large relative to its cost, which is a few percent of the total. It also fits neatly with the warmup-stable-decay learning rate schedule, whose long constant-rate middle allows training to be extended or stopped at will, with the expensive decay-and-anneal performed once at whatever endpoint is chosen. And it has quietly moved the boundary between pretraining and post-training: capabilities that were previously introduced by fine-tuning are now increasingly folded into the end of pretraining, where they cost less and stick better.
Figure 3. The bulk phase, then the last week.
What to carry forward
A model is largely made of what it read, and the decisions governing that are the least documented and possibly the most consequential in the pipeline. Raw web text survives filtering at a rate of a few percent, through cheap heuristics first, then deduplication — which improves models both by preventing wasted capacity on repeated text and by keeping evaluations uncontaminated — and finally quality classification against a reference corpus, a step that unavoidably encodes somebody’s judgement about what good writing is. Of the mixing proportions, two things are reliably known: a substantial share of code improves reasoning on tasks unrelated to code, and the right filtering strictness depends on the compute budget, since aggressive filtering is correct only when you will not be forced to repeat the survivors. Repetition itself is nearly free to about four passes, still worthwhile to around sixteen, and worthless by forty — which multiplies the effective supply of human text without removing the limit. Synthetic data addresses the shortage and works when paired with an external check, such as a test suite or a stronger teacher; the failure mode requiring care is a closed loop in which real data is replaced by a model’s own samples, which narrows the distribution generation by generation as the tails go unsampled. And ordering matters most at the end: a final annealing phase on the highest-quality material, as the learning rate decays to zero, buys a disproportionate share of the model’s quality for a few percent of its cost, and has been steadily absorbing work that used to belong to fine-tuning.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
