Almost everything else in the machinery of a language model can be verified. A gradient is correct or it is not; a cache holds the right keys or it does not; a quantized model produces outputs close to the original or it does not. Evaluation is the one activity where the ground keeps moving. We build models to be generally capable, then measure them with tests that are necessarily specific; the tests leak into the training data; the numbers rise while the thing being measured may not have improved; and the community moves to a new test and repeats the cycle. None of this is fraud, and most of it is not even carelessness — it is the predictable consequence of optimizing against a measurement. This piece is about why evaluation is genuinely hard, which failures are structural, and what can actually be trusted.
What makes it hard
Four difficulties, and they are different in kind rather than degrees of the same problem.
The target is unbounded. A model is meant to be good at an open set of tasks that nobody has enumerated. Any benchmark samples from that set, and a sample is only as representative as its designer’s imagination. Excellent performance on the sample supports a conclusion about the sample.
Loss is not capability. As the article on scaling laws noted, the smooth quantity we can measure precisely is cross-entropy on held-out text, and the mapping from that to whether a model can do anything useful is neither smooth nor known.
Most outputs have no right answer. Multiple choice can be scored. A summary, an explanation, a piece of advice, a paragraph of prose — these have quality rather than correctness, and quality requires a judge.
The measurement changes the thing measured. This is the one that dominates everything else. Once a benchmark matters, it is optimized against, directly through training data selection and indirectly through a thousand small decisions. A measure that becomes a target stops measuring what it did.
Contamination
This is the most concrete of the problems, and the most widespread.
Models are trained on text scraped from the internet. Benchmarks are published on the internet, discussed on the internet, reproduced in papers, quoted in blog posts, and included in datasets of collected evaluations. A model that has seen the test items during pretraining is not being tested; it is being asked to recall.
Contamination is rarely deliberate. It arrives through mirrors nobody knew about, through aggregator datasets that silently include benchmark suites, through discussions that quote items verbatim, and through synthetic data generated by a model that was itself contaminated. Deduplication helps and does not solve it, because a paraphrased or translated item does not match by hash while conveying the same answer.
Detection is imperfect. Comparing a model’s perplexity on benchmark items against similar unseen text is suggestive — memorized text is unusually unsurprising — and so is checking whether a model can complete an item from its first few words alone, or whether it performs better on the canonical ordering of multiple-choice options than on a shuffled one. None is conclusive.
The strongest defences are procedural rather than statistical. Hold out a private test set and never publish it. Use benchmarks constructed from material published after a model’s training cutoff, which is the cleanest possible guarantee and requires continuously generating new ones. Report on tasks whose answers are computed rather than looked up. The general shape of the answer is that a benchmark’s useful life is finite from the moment it is published, and that treating benchmarks as durable is the mistake.
Judging what cannot be scored
For open-ended output, someone has to judge. There are three options and each is compromised.
Humans set the reference standard and charge for it. An annotator costs money, takes days, and disagrees with the annotator beside them: agreement on subjective quality is often mediocre, and human raters have documented biases — toward longer answers, toward confident phrasing, toward fluency over accuracy. The same biases the article on preference learning identified as sources of reward hacking are present in the evaluation, because it is the same signal.
Reference-based automatic metrics compare output against a written answer by string overlap. They cost almost nothing, reproduce exactly, and measure the wrong quantity: a correct answer phrased differently scores badly, and a wrong answer sharing vocabulary scores well. They persist mainly through inertia.
Model judges are now the dominant approach: present a stronger model with the output and a rubric and ask for a score or a comparison. It costs little, runs fast, and tracks human judgement better than string metrics do. It also carries specific and well-documented biases. Judges prefer longer responses. They prefer the first option presented, so pairwise comparisons must be run in both orders and averaged. They prefer text resembling their own generation style, which means a judge from the same family as one contestant is not neutral. And they are poor at detecting errors they would have made themselves, which puts a ceiling on using a model to evaluate anything near its own competence.
None of this makes model judging useless. It makes it an instrument with known distortions, to be calibrated against human judgement on a sample rather than trusted absolutely.
Figure 1. Three judges, three sets of distortions.
What can actually be trusted
Some measurements resist the failures above, and the pattern connecting them is worth extracting.
Executable evaluation. Where correctness is checked by running something — code against tests, a proof through a checker, a query against a database — the measure is a fact rather than a judgement. This is the same property that makes verifiable rewards work, and it makes an evaluation similarly hard to game. It does not prevent contamination of the problems, but it does mean partial credit and stylistic flattery cannot be earned.
Held-out and freshly-generated tests. Problems built after the training cutoff, or procedurally generated from templates so that no specific instance was ever published, are contamination-proof by construction.
Head-to-head comparison with real users. Serving two anonymous models to people who came with their own tasks, and collecting preferences, avoids the fixed-question problem entirely and cannot be trained against directly. Its weaknesses are the human biases above and a population that is not representative of any particular use.
Adversarial and perturbation testing. Rephrase a question, permute the answer options, change the names and numbers. A model that understood the task is robust to these; a model that memorized the item is not. The gap between original and perturbed performance is one of the more honest signals available, and it is reported far too rarely.
A pinch of math: how many samples the number needs
One quantitative point, because benchmark differences are compared far more casually than the sample sizes justify.
Scoring $n$ items with a per-item success probability $p$ gives an observed accuracy whose standard error is
$$\text{SE} = \sqrt{\frac{p(1-p)}{n}}$$For a benchmark of 500 items at around 70 percent accuracy, this is about $0.020$ — two percentage points. A 95 percent confidence interval spans roughly four points, and comparing two models requires accounting for the uncertainty in both, so a difference smaller than about four or five points on such a benchmark is not distinguishable from noise.
This is before considering the other sources of variance, which are usually larger: the choice of prompt template can move a score by several points, the number of examples shown in the prompt matters, the decoding settings matter, and the answer-extraction logic matters. Published tables routinely report differences of one or two points as meaningful. Most of them are not, and the practice of reporting a single number with no interval is the field’s most common statistical failing.
Figure 2. The error bars a benchmark number omits.
What to carry forward
Evaluation is hard for reasons that do not go away with effort: the capability being measured is unbounded while any test is a sample of it; the loss we can measure precisely does not map smoothly onto usefulness; most valuable outputs have quality rather than correctness; and any measure that comes to matter is optimized against and stops measuring what it did. Contamination is the concrete face of this — benchmarks are published on the internet and models are trained on the internet, so a test item may be recalled rather than solved, and it arrives through mirrors, aggregators, discussions, and synthetic data rather than through deliberate cheating. Detection is suggestive rather than conclusive, so the real defences are procedural: private held-out sets, problems drawn from material published after the training cutoff, and answers that are computed rather than looked up. For open-ended output a judge is unavoidable, and all three kinds are compromised — humans are inconsistent and biased toward length and confidence, string-overlap metrics measure the wrong thing entirely, and model judges, now dominant, favour long answers, whichever option is presented first, text resembling their own style, and are blind to errors they would have made themselves. What survives is evaluation that is executable, where a program checks the answer; tests generated fresh or held private; head-to-head comparison on tasks real users brought; and perturbation testing, where rephrasing a question or permuting its options separates understanding from memorization. And the numbers deserve error bars: a five-hundred-item benchmark carries a standard error around two points before prompt-format variance is counted, which makes the routine reporting of one- and two-point differences as meaningful the field’s most common statistical failing.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
