Memorization arrived in machine learning with a settled meaning and a verdict attached: a model that had fitted its training set too closely, which was bad and, better still, detectable — watch the held-out loss, and when it turns upward while the training loss keeps falling, stop. That diagnostic is still taught, still correct on its own terms, and almost beside the point for a model trained once over a corpus it will never exhaust. The held-out loss does not turn upward. Nothing on the dashboard looks wrong. The model has nonetheless retained enough of its training data to reproduce parts of it word for word, including passages that appeared a handful of times and strings nobody wanted kept. The old alarm stays quiet because it was wired for a different fault. This piece is about why the training objective asks for verbatim retention rather than merely permitting it, what determines which sequences receive it, and what the available remedies do and do not remove.
Nothing in the objective distinguishes a fact from a string
Training a language model consists of raising the probability it assigns to text that actually occurred, position by position, measured by cross-entropy . Take the sentence the capital of France is Paris. Two different changes will reduce the loss on it. One is to acquire something general: a representation of countries and their capitals, wired into the syntax of that construction, which pays off again on Lisbon and Ottawa and every sentence shaped like this one. The other is to store the sequence — to arrange the weights so that this prefix makes this continuation likely, with no consequence anywhere else.
Both lower the loss. Gradient descent sees the loss and nothing else. It has no term expressing a preference for the general solution, no penalty on the specific one, no representation of the distinction at all; generalization happens only because the general solution is usually cheaper in parameters than storing every instance, and where it is not cheaper, storage wins. So memorization is not a defect in the pretraining objective . It is one of the things the objective asks for.
Memorization is necessary, not merely tolerated. This cuts against the reflex, so it belongs first. A model must retain a great deal verbatim to be useful at all. Idioms are not derivable, names attach to people by convention, code syntax is a set of arbitrary strings that must be reproduced exactly or the program will not run, and quotations are quotations. Spelling is a fact about a word, and given how tokenization chops text into fragments, even ordinary orthography must be partly held rather than inferred. A model that memorized nothing would know nothing in particular. The question is never whether a model memorizes, but what, and which of it should not have been kept.
Figure 1. Two ways to lower the loss on one sentence.
What predicts whether a sequence is kept
Retention is not uniform across a corpus, and the factors governing it can be ranked.
Duplication dominates everything else. A sequence appearing many times in the training data is far more likely to be reproducible than one appearing once, and the relationship is steep enough that reducing duplicate counts moves reproducibility more than any other lever. It is why deduplication, as work on data mixtures reports, is the cheapest effective intervention. Every additional copy is another pass over the same tokens, applying the same pressure in the same direction.
Model size pushes the wrong way. At equal exposure, a larger model retains more. Capacity is what storage consumes, so this is unsurprising, and it sits awkwardly beside the scaling results, because the extra parameters that buy better generalization also buy more literal retention. They are both what capacity is for.
Distinctiveness is the uncomfortable factor. A bland sentence is protected by its own predictability: general competence already assigns it decent probability, so little residual loss is left for storage to claim. A random-looking string enjoys no such protection. No general pattern would produce it, so the only route to lowering its loss is to raise the probability of those tokens after that prefix, which is storage by definition. The sequences most exposed are therefore the ones that look like identifiers — keys, account numbers, unusual names, anything whose whole point is that it could not be guessed.
Retention is measured by asking a trained model to continue text it was trained on. That is the whole mechanism of extraction as a research technique, and the detail worth giving here is the detail a security paper gives a vulnerability class: the mechanism, without the recipe. The shape of the findings is what matters. A non-trivial quantity of training data is recoverable from a deployed model, the quantity grows with model size, and the recoverable material skews toward duplicated or distinctive text. Extraction measures what is already present. It does not put it there.
Figure 2. The three factors that decide what is kept.
A pinch of math: why an incompressible string has to be stored
The argument is short and it explains the distinctiveness effect. For a sequence of $k$ tokens, the loss contributed is
$$L_{\text{seq}} = -\sum_{t=1}^{k} \log p(x_t \mid x_{Exposure follows the same accounting. A sequence appearing $n$ times contributes $n$ times the gradient pressure toward whatever reduces its loss, so where that reduction can only come from storage, duplication buys storage linearly and no general capability at all. Repeated material is not thereby more worth generalizing. It is only pushed harder.
Against this sits a hard ceiling. A model has finitely many parameters, each holding finitely many bits, so a corpus much larger than the model cannot fit inside it, and the bulk of training data is therefore not recoverable at all. The bound is real and nearly useless, because it constrains the total without constraining the selection. Knowing that a vault holds a thousandth of the documents put into it tells you nothing about which thousandth.
The overfitting diagnostic is watching a different quantity
In the single-pass regime the classical alarm is silent. Held-out loss falls throughout and never turns upward, the train-validation gap stays narrow, and the curve reads as generalization rather than noise-fitting — while rare sequences are retained verbatim. Both are true at once, because they are separate phenomena. The classical failure is a model fitting a finite sample so tightly that its predictions elsewhere degrade, which the diagnostic sees as lost average performance. Verbatim retention of one distinctive string costs no average performance. It is local, and leaves behaviour everywhere else intact, which is why an aggregate measure cannot see it. The curve is not lying. It is answering a question nobody asked here.
The same disconnection undermines the reflex to reach for regularization . Weight decay, dropout, and early stopping were built against the classical failure: they constrain how idiosyncratic the fit is on average, and do little about the capacity to hold one improbable string. Nothing in the toolkit suppresses targeted retention while preserving the retention that makes a model useful — unsurprising, since the objective cannot tell the two apart either.
Figure 3. The overfitting alarm is wired for a different fault.
Where the retention does damage, and what each remedy costs
Three consequences, of different kinds, and then the interventions.
Contamination makes evaluation unreliable in a way that is nearly impossible to rule out. A benchmark item present in the training corpus is not being answered; it is being recalled. Once the corpus is web-scale this is very hard to exclude, because items propagate through mirrors, aggregated collections, papers quoting them, and synthetic text from an already-contaminated model. The consequence for evaluation is not that scores are wrong but that their error is unmeasurable, which is worse.
Privacy is damaged by reproducibility rather than by secrecy. Most of the material at issue was public when collected. What changes is that a document in an obscure corner of the web, findable by someone who knew to look, becomes something a deployed system can emit on demand to someone who did not. The distinction that matters is between available and redistributable, and a model trained on public text quietly converts the first into the second.
The copyright question is contested and legally unsettled. Whether training on protected work infringes, and whether verbatim reproduction changes the analysis, are live disputes varying by jurisdiction, and nothing here adjudicates them. The technical statement is narrower: verbatim reproducibility is a measurable property of a trained model, its extent depends on duplication and distinctiveness, and it is therefore something a pipeline can influence.
Every available intervention buys less than it appears to. Deduplication is cheap, effective, and blind to the near-duplicates that matter most, since a document reformatted, translated, or paraphrased collides with nothing while applying the same pressure. Filtering identifiers and secrets out of the corpus works to the extent the detectors do, which for structured credentials is decent and elsewhere is hope. Differentially private training bounds how much any single example can influence the result — a genuine guarantee, at a quality cost large enough to keep it rare at this scale. Output-side blocking, during decoding or afterwards, catches the exact string and not the paraphrase.
None of these removes what is already in the weights. Unlearning a specific fact from a trained model remains unsolved. The methods that exist suppress the behaviour — the model declines or fails to produce the sequence — while leaving open whether the information is still present and reachable by another route, and ablation studies find suppression far more often than removal. The thinness is familiar from interpretability work: as the study of superposition shows, information here is spread across directions corresponding to nothing a surgeon could excise, so a request to delete one item does not name a place.
What to carry forward
The training objective grades one thing, the probability assigned to text that actually occurred, and cannot distinguish a fact worth generalizing from a string worth reciting: both reduce the loss, and gradient descent sees only the loss. That makes memorization something the objective asks for rather than a failure of it, and the corollary is that much verbatim retention is wanted: idioms, names, code syntax, quotations, and spelling are derivable from nothing, and a model that kept none of them would know nothing in particular. The live question is which retained material should not have been kept, and three factors govern it. Duplication dominates, steeply enough that deduplication is the most effective intervention available. Larger models retain more at equal exposure. And distinctiveness exposes the wrong material, because a bland sentence is already probable under general competence while a random-looking string can only have its loss reduced by storing it, which makes identifiers and keys the most exposed text in any corpus. Research extraction demonstrates this rather than creating it, reporting a non-trivial, size-dependent quantity of recoverable data skewed toward duplicated and distinctive text. The familiar overfitting curve reports nothing wrong throughout, since targeted retention costs no average performance on unseen text, and the regularizers built for the classical failure do not touch it. The damage lands on benchmark validity, on the conversion of available material into redistributable material, and on a copyright question that is unsettled. Deduplication misses near-duplicates, secret filtering is only as good as its detectors, private training is expensive enough to stay rare, and output blocking handles the string and not the paraphrase. All of them act before or after the weights. Removing a specific memorized item from a trained model is not something anyone knows how to do.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
