The limit on human preference data is usually presented as a matter of money: annotation is expensive, budgets are finite, so the field went shopping for something cheaper. That framing gets the constraint wrong. What limits preference data is not its price but the fact that somebody has to read two long responses attentively enough to have a defensible opinion about which is better, and attention of that kind does not scale and cannot be bought in bulk. A second limit sits behind it: an annotator cannot demonstrate a good answer to a question they cannot answer, so on difficult material the ceiling is competence rather than diligence. The field’s answer was to hand the comparison to a model. This piece is about what that substitution leaves untouched, what it makes newly legible, and the moment its errors stop cancelling.
The constraint is arithmetic, not sentiment
Put numbers on it, loosely. A preference dataset that meaningfully moves a model’s behaviour wants comparisons in the tens or hundreds of thousands. Each is a prompt and two responses of a few hundred words, and a person who skims them produces a label worse than no label, because noise in preference data is indistinguishable from taste. Budget a few minutes per pair and the arithmetic turns unpleasant: a hundred thousand comparisons is thousands of hours of undivided attention, the one input in the training stack whose unit cost has never fallen.
Then multiply, because helpfulness is only the first axis. Tone is another, and it contradicts helpfulness at the margins. Format, refusal behaviour , factual care, ambiguous requests, the right quantity of hedging — each is a separate property, wanting comparisons drawn from prompts that exercise it. The demand is a product rather than a sum. The bottleneck is not enthusiasm, and not really money either. It is hours.
The move that follows is unromantic. Show a capable model the same prompt and the same two responses, with a written statement of what to prefer, and record whichever it picks as the label. Nothing downstream changes. The pairs feed a reward model exactly as human pairs did, or go straight into direct preference optimization , and the policy optimization consuming the result cannot tell where they came from. It is easy to mistake this for a new algorithm. It is a change of data source. Everything already true of preference learning — the reward is a proxy, the proxy gets gamed, the leash to the reference model is load-bearing — remains true, with one fewer person in the room.
Figure 1. The same pipeline, one fewer person in the room.
The written principles are the interesting part
This is where the substitution earns more than a line in a budget, and the reason is legibility rather than cost. Human preference data encodes its criteria implicitly. What the annotators were rewarding lives distributed across a hundred thousand judgements, each a single bit, none with a reason attached. To find out what such a dataset asked for, you fit something to it and observe what the result favours. That is an experiment, not a reading.
A written statement of what to prefer is a different kind of object. It is explicit, it sits in version control, and it can be argued with by people who never saw a label. You can read the line saying brevity is preferred where no detail is lost, decide it is producing answers that omit caveats, change that line, relabel, retrain, and compare. That is a real gain, and it deserves stating before it is qualified.
Here is the qualification. What has become legible is the instruction, not the behaviour. The document is an input to the judge’s reading of a pair, and that reading is a forward pass through a network with habits of its own. The gap between what the principles say and what the model does is what the procedure both closes and fails to close: closed enough that the document is worth writing, open enough that reading it tells you what was asked for rather than what was learned. An auditable specification attached to unaudited behaviour beats no specification. It is not a guarantee, though it is occasionally taken for one.
Judging is easier than producing, and only up to a point
That the labels carry information rests on an asymmetry worth stating carefully. Deciding which of two answers is better is strictly easier than writing the better answer. A reader who could not have derived a proof can still notice that one attempt skips a case. So a model supplies useful signal about outputs it could not have produced — the same asymmetry that makes checking cheaper than generating in verifiable rewards , where a program that could never write the code under test can still run it, and the reason sampling many candidates and picking among them beats spending the same compute on one.
The asymmetry is real and bounded, and the boundary sits somewhere inconvenient. It holds where the flaw is visible in the response: a contradiction, an omission, an evasion, a mess. It collapses wherever recognizing the better answer needs knowledge the judge lacks. A fabricated citation with a plausible author and year reads as scholarship to a judge that has not read the literature, and the confident wrong answer beats the hedged right one on every axis available — much of the mechanism behind hallucination surviving training. That is the territory where the human ceiling hurt most, and the model ceiling sits in the same place.
An unsettled argument rests on top of this, better named than resolved. One reading treats model feedback as a new source of signal: the judge does work at label time that nobody did before. The other treats it as distillation — the judge’s preferences amortized into a second model at volume, the written principles serving as a prompt rather than as information. Which reading applies depends on the arrangement. Where the judge is much stronger than the model being trained, distillation explains most of what happens and predicts the ceiling correctly. Where the judge is the same model, or weaker, distillation explains nothing and the asymmetry does all the work. Both are in use, which is much of why the argument persists.
Figure 2. Where judging stops being easier than producing.
A pinch of math: what averaging annotators was buying
The cleanest formal statement of the cost is a decomposition, and it explains what a crowd was for. Write one labeller’s error on one comparison as a sum of two parts:
$$\varepsilon_i = b + v_i$$Here $b$ is the bias shared by everyone labelling, a systematic lean toward longer responses, say, and $v_i$ is idiosyncratic noise, independent across labellers, with mean zero and variance $\sigma^2$. Collect $n$ labels for one comparison and average them. The shared part survives untouched, while the idiosyncratic part becomes $\frac{1}{n}\sum_i v_i$, leaving
$$\mathbb{E}\big[\bar{\varepsilon}^{\,2}\big] = b^2 + \frac{\sigma^2}{n}$$The second term shrinks as $1/n$, so its typical size shrinks as $1/\sqrt{n}$, and can be pushed as low as the budget allows. The first term does not move. Total error does not tend to zero as labellers are added; it tends to $b^2$.
Read both arrangements against that. A pool of human annotators was expensive because it was buying reductions in $v$, and it worked: human $v$ is large while human $b$ is small and pulls in several directions at once — one rater’s systematic lean is, across a diverse pool, part of somebody else’s noise. A single model inverts both halves. Reducing $v$ becomes nearly free, since the judge can be sampled repeatedly or run cool. And $b$ becomes irreducible, because there is one labeller and every comparison inherits the same lean. The rounding error is now the whole of the error.
Figure 3. Error floors at the judge’s bias, not zero.
What the substitution costs
Three consequences follow, and together they make this a trade rather than a saving.
The errors correlate. Ten annotators who each misjudge a different tenth of the pairs produce scattered, partly self-cancelling noise. One model that misjudges a particular kind of pair misjudges every instance of that kind, in the same direction, a million times. Its blind spots are not averaged away; they are stamped uniformly into the reward signal, indistinguishable from preferences and learned as such. The failure is not that the model is a worse judge than a person, which it often is not. The failure is that its mistakes arrive in perfect agreement with each other.
The judge has documented biases. It prefers longer answers. It prefers confident phrasing to hedged phrasing. It prefers whichever option came first, so pairs must be run in both orders and averaged. It prefers text resembling its own style, which makes a judge from one candidate’s own family a partisan rather than an arbiter. That is the list that turns up whenever models are used as graders, as the article on evaluation sets out, and it is the same list because it is the same act. An evaluation with known distortions can be corrected for; a training signal with known distortions becomes the thing optimized. The bias stops being measurement error and becomes the objective.
It is a closed loop. A model producing the data that trains the next model has no external reference, and preferences nobody wrote down compound across rounds into behaviour nobody chose — a register that drifts, a house style that hardens, a category of question that stops being answered. Detecting that means reading outputs and judging them, which is the human evaluation the arrangement was built to reduce. The saving is partly a transfer rather than an elimination: hours move out of labelling and into auditing, at a better exchange rate but not a free one.
Which is why hybrids dominate practice, and why that is the honest empirical position rather than a compromise. Model labels cover volume, supplying the comparisons no budget would otherwise reach. Human labels are kept for calibration — a sample scored both ways, to see where the judge agrees with people and where it does not — and for the cases nobody will let a model decide, usually the safety boundary and whatever the last audit found going wrong.
What to carry forward
Preference training wants a great many comparisons, each requiring a person to read two responses with real attention, and attention of that kind has never got cheaper; multiply by every axis worth training — helpfulness, tone, format, refusal, factual care — and the demand becomes a product rather than a sum. Handing the comparison to a capable model, guided by a written statement of what to prefer, relieves that without altering anything downstream: the labels feed a reward model or a direct preference objective exactly as human labels did, so this is a change of data source, and every existing property of preference learning survives intact. The written principles are the real gain, because criteria that previously existed only as a distribution over thousands of unexplained judgements become a document you can read, dispute, and amend one line at a time — though what is legible is the instruction rather than the behaviour, and that distance is not closed by writing the instruction more carefully. The labels carry information because judging is easier than producing, the asymmetry that makes checking cheaper than generating elsewhere; it is bounded, failing wherever recognizing the better answer needs knowledge the judge lacks, which is where help was most wanted. The costs are structural. Independent human errors partly cancel and one judge’s errors do not, so total error tends to a floor set by the judge’s bias rather than toward zero. Its documented leanings — length, confidence, position, its own style — become the objective. And a model generating the data for the next model can drift somewhere nobody chose, detectable only by the human reading the arrangement was meant to avoid. What is bought is volume. What is spent is the diversity of error that made aggregation work at all.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
