Training a language model is, up to a point, mechanical: choose a loss, differentiate it, descend. Each stage works because it has something concrete to compare against — the next token in the corpus, a demonstration written by a person, the verdict of a test suite. Then the requirement becomes be helpful, and the machinery has nothing to bite on. Nobody can write a differentiable expression whose minimum is a tactful reply, or one that prefers the right amount of hedging to slightly too much. The properties are real; they have no closed form. What people can do is look at two answers to the same question and say which is better, often without being able to say why. Preference-based alignment rests on one substitution made at that point: fit a function to the comparisons, then treat the function as the objective. This piece is about how that function is built, what the substitution buys, and the ways it goes wrong.
Ratings drift, comparisons hold still
The obvious way to gather data about quality is to ask for it directly: show an annotator a response, ask for a score out of ten, fit a regressor to a hundred thousand of them. It works badly, for reasons that have nothing to do with the model. A seven means different things to different people, because each annotator is privately inventing the scale as they use it. Worse, the scale moves. Two hours of mediocre answers recalibrates a rater without their noticing, so a response that earned a six in the first hour earns an eight in the third, and the drift leaves no trace.
A comparison asks for much less. The annotator never has to locate an answer on an absolute scale, only to break a tie against a reference sitting on the screen. That judgement is markedly more stable, between people and within a long session, because most of what corrupts an absolute rating is a property of the rater’s state — and it corrupts both members of a pair identically, so it cancels.
The cost is information. A comparison carries close to one bit where a rating carried a number, so far more are needed, and it records order without distance: a marginally cleaner answer and an unusable one produce the identical annotation. And a coin flip on an indistinguishable pair enters the data looking exactly like a preference.
The mechanism is the model with its head replaced
The construction is unromantic. Take a language model, discard the unembedding matrix that would have projected the final hidden state onto the vocabulary, and bolt on a single linear head producing one number. Run the prompt and the whole response through it, read the scalar at the last position, and that is the reward. The same weights that were reading text to predict tokens are now reading it to produce a score.
Initialization matters more than it looks. The head is attached to the supervised fine-tuned model rather than to a base model, because a reward model started from scratch would have to learn what English is before it could learn what a good answer is, and a hundred thousand preference pairs cannot teach both. The head reads off a distinction the model can already draw.
Training is a logistic loss over the pairs, run for very few passes, since preference data is scarce and these models overfit within an epoch or two. And the scale is arbitrary. Add a constant to every score and the loss is unchanged; nothing pins down where zero sits. Only differences mean anything. It is the situation of a logit , meaningless in isolation while a difference of two logits is an honest log-odds — so a reward cannot be compared across models or runs, or read as a grade.
A pinch of math: the logistic model of a preference
To fit a function to comparisons we need a rule turning two scores into the probability a person picks one. The standard choice is the Bradley–Terry model, borrowed from the statistics of paired comparisons long before anyone applied it to text. For a prompt $x$, a preferred response $y_w$ and a rejected one $y_l$, it posits:
$$P(y_w \succ y_l \mid x) = \sigma\big(r(x, y_w) - r(x, y_l)\big), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}$$Each piece is there for a reason. The subtraction is there because a preference is a comparison and nothing else was ever observed. The logistic $\sigma$ squashes an unbounded real difference into a probability, and behaves sensibly at the boundaries: equal scores give one half, a large gap near-certainty, and the transition between is differentiable. Maximum likelihood over such triples gives the loss the apparatus is trained with,
$$\mathcal{L} = -\log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big)$$a two-outcome cross-entropy, structurally identical to what any binary classifier minimizes. Two things fall out of it.
The reward is identified only up to an additive constant per prompt. Replace $r(x,y)$ by $r(x,y) + c(x)$ for any function $c$ of the prompt alone, and every within-prompt difference is untouched, so the loss sits where it was. The model never learns an absolute standard: its scores compare two answers to one question and nothing across questions, which is why every downstream use supplies its own reference point, and why policy optimization leans on baselines and advantage whitening.
And the gradient separates a pair by as much as it is currently wrong. Writing $\Delta = r_w - r_l$, the derivative of the loss with respect to the margin is $-\big(1 - \sigma(\Delta)\big)$, so the scores are pushed apart in proportion to the probability the model assigns to the wrong ordering. A pair already ranked correctly by a comfortable margin contributes almost nothing; a pair it gets confidently backwards dominates. Training concentrates on its own mistakes, and nothing bounds the margin: where every annotator agrees, the loss keeps paying for a wider gap forever. That is why the run is kept short.
Figure 1. Bradley–Terry, and where the gradient goes.
Figure 2. Only differences within a prompt mean anything.
What the substitution costs
Three costs follow from fitting a function to preferences and then optimizing it, and they differ in kind.
It is a proxy, and proxies get gamed. The reward model learned correlations from a finite pile of comparisons, some tracking quality, some accidents of collection. Longer answers scored slightly better on average, so length becomes a lever. Bulleted structure looked thorough, so everything arrives in bullets. Agreement scored better than contradiction, so the model becomes agreeable — sycophancy is not a character flaw picked up somewhere but a correlation in the labels, exploited as instructed. Confident phrasing outscored hedged phrasing, so hedging thins out whether or not there are grounds. None of this is misbehaviour. It is maximizing the number it was given.
It is only valid near the data it was trained on. The fitted function is accurate where comparisons were collected and unconstrained everywhere else. As the policy improves it leaves that region for territory the reward model has never scored, where a smooth surface pushed hard tends to have spurious peaks. So the reward becomes least trustworthy at the moment training is going well. This is why a KL penalty to the reference model exists at all: not a refinement but a fence against travelling to where its judge cannot see. The fence also caps how much can be learned, which is the bind the method never escapes.
Agreement between annotators is itself mediocre. On subjective quality, two people shown the same pair often agree around seventy percent of the time, and that is a ceiling rather than an obstacle to be engineered away: no model can predict a judgement the judges do not share. A reported accuracy in the low seventies is therefore at the noise floor, not a result. Worse, the biases in human raters — toward length, confidence, and fluency over accuracy — are the same ones described above as reward hacking, which the article on evaluation reaches from the other side. The hacking is not a distortion introduced by the fitting. It is faithful transmission of a distortion already in the source.
Figure 3. Rater bias in, reward hacking out.
What it cannot say, and the two escapes
Beyond the costs, there are two things a reward model does not do at all, and neither is a matter of accuracy.
It scores a whole response with one number. A five-hundred-token answer with one confused paragraph and four good ones receives a single scalar, and nothing in it says which paragraph was the problem. The reinforcement learning downstream spreads that verdict across every token, so a mostly-good response is punished uniformly and a mostly-bad one rewarded uniformly, leaving the model to infer where the error was from many attempts. Process-level scoring, grading each step of a derivation, exists to address this, and it needs a labeller for those steps — people, which is expensive, or a model, which reintroduces the proxy.
It cannot represent disagreement. Where people genuinely differ — how blunt an answer should be, how much a difficult topic should be softened, where a refusal belongs — the fitting does not record a split. It averages. The result is a target between two incompatible positions, corresponding to nobody’s actual preference, and a policy optimized against it converges on a compromise no annotator would have chosen. The honest object would be a distribution over rewards, or several models for several populations. A single scalar has no room to carry the disagreement, so it is quietly deleted.
Two escapes exist, with limits worth stating. The first removes the model: where an answer can be checked by a program, verifiable rewards substitute a test suite or a proof checker for the learned judgement, and the reward becomes a fact that cannot be flattered. The second removes the separate training stage: direct preference optimization reads the reward off the policy itself as a log-ratio against its starting point, deleting the network without deleting the reliance on preferences. Neither reaches the territory that motivated the construction. For everything no program can check, comparisons are the only instrument available, and a function fitted to them the only objective on offer.
What to carry forward
The reward model exists because there is no loss function for a good answer: helpfulness, tone, tact and appropriate hedging are properties with no closed form, while a person shown two responses can say which is better without being able to say why. Every problem descends from the substitution: fit a function to the comparisons, then optimize it. Comparisons rather than absolute scores, because ratings are badly calibrated between annotators and drift within a session, whereas what corrupts a rating usually corrupts both members of a pair and cancels in the choice; the price is a label carrying roughly one bit, recording order without distance. The construction is the language model with its unembedding replaced by a scalar head, initialized from the supervised fine-tuned model, and trained briefly with a logistic loss under the Bradley–Terry assumption that a preference is a logistic function of a difference of rewards. Two properties follow: the reward is fixed only up to an additive constant per prompt, so scores compare within a question but not across questions, and every downstream use supplies its own baseline; and the gradient separates the two scores in proportion to how confidently the current ordering is wrong, with nothing bounding the margin, which is why the run is kept short. The costs are that a proxy gets gamed — length inflation, formulaic structure, sycophancy, confident phrasing displacing warranted uncertainty; that its validity is local, so the reward degrades as the policy improves and moves off the data, which is why the KL fence is load-bearing; and that annotator agreement near seventy percent caps achievable accuracy, making a reported figure at that level the noise floor, not a result. What it never does is say which part of a response was bad, or admit that people disagree.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
