Almost every model in wide use has learned by being shown what a good answer looks like. In pretraining the good answer is simply the next token in the corpus. In supervised fine-tuning it is a demonstration written by a person. In preference learning it is whichever of two responses a human preferred — or, more precisely, whichever a reward model, itself trained to imitate those humans, predicts a human would have preferred. That last substitution is a compromise the field has quietly lived with for years: the thing doing the grading is a neural network with opinions and blind spots of its own. Reinforcement learning from verifiable rewards, usually shortened to RLVR, is what becomes possible when you can throw the grader away and simply check the answer. This piece is about what changes when the reward stops being a judgement and starts being a fact.
The problem with learning from a judge you also trained
Preference learning works roughly like this. Collect pairs of model responses, ask people which they prefer, train a smaller model to predict those preferences, and then use that model’s score as a reward while the main model generates. The reward model is a stand-in for human taste, and it has to be, because you cannot put a person in the loop for the millions of judgements that reinforcement learning consumes.
The trouble is that a stand-in can be satisfied without the original being satisfied. The reward model has learned correlations from a finite set of comparisons, and some of those correlations are real signals about quality while others are accidents of how the data was gathered. Longer answers scored slightly better on average, so length becomes a lever. Confident phrasing scored better than hedged phrasing, so hedging disappears whether or not the model has grounds for confidence. Bulleted structure looked thorough, so everything arrives in bullets. None of this is deception on the model’s part; it is doing exactly what we asked, which is to maximize a number. It is our number that was wrong.
This is reward hacking, and it is not an occasional malfunction but the expected outcome of pushing hard on an imperfect proxy. Which is precisely why the standard recipe includes a leash: a penalty on how far the model’s output distribution is allowed to drift from where it started, so that it cannot wander into the strange regions of language where the reward model’s scores are highest and its judgement is worst. The leash keeps things safe, and it also caps how much can be learned. You are permitted only small departures from the model you began with, because large departures are exactly where the proxy stops being trustworthy.
So the ceiling on preference learning is not really about the algorithm. It is that we do not have a reward we can push on indefinitely.
A reward you cannot flatter
For some questions, we do. The answer to a competition mathematics problem is a specific number, and a short program can compare the model’s final answer against it. A function is supposed to sort a list, and you can run it against a hundred test cases. A proof written in Lean either typechecks or it does not. A generated SQL query either returns the expected rows or it returns something else. A puzzle either satisfies its constraints or it violates one.
In each of these the reward is not an estimate of quality. It is a fact about the world, produced by a verifier — a piece of ordinary software, deterministic, cheap to run, with no parameters and no opinions. A unit test cannot be charmed by a confident tone. A theorem prover does not find bulleted structure persuasive. The lock either opens or it does not.
This changes the economics of the whole procedure. Because the reward is honest, you can optimize against it far more aggressively than you would ever dare against a learned one. The leash can be loosened or, in some setups, removed entirely, and the model is free to travel a long way from where it started. It can develop habits that no human demonstration contained, and as long as those habits raise the fraction of locks that open, they will be reinforced. The one bit of information — right or wrong — is thin, but it is incorruptible, and it turns out that incorruptible beats rich.
Figure 1. A learned judge on a short leash; a checker on none.
What the model is actually being taught
Here is the part worth sitting with, because it is genuinely strange.
The model produces a long chain of reasoning and then an answer. The verifier looks only at the answer. Nothing in the training signal says anything whatsoever about the reasoning — not whether a step was valid, not whether the approach was sensible, not whether it resembles how a person would work. The reward arrives at the very end, attaches to the entire trajectory, and is silent about every token that produced it.
This is outcome supervision, and it should by rights be far too crude to work. Imagine grading a thousand-word derivation by writing a single tick or cross at the bottom and handing it back. And yet what emerges from it, without anyone specifying it, is a repertoire of things that look remarkably like thinking: allocating more tokens to harder problems, checking an intermediate result before continuing, noticing an inconsistency and going back, trying a second approach when the first stalls. DeepSeek’s report on R1-Zero — a model trained with RLVR directly on a base model, with no supervised demonstrations of reasoning at all — includes a now-famous passage where the model interrupts itself mid-derivation to reconsider. Nobody wrote that behaviour down for it to copy. It appeared because chains that pause to check are, on average, chains that end at the right number.
The mechanism behind that emergence is worth being precise about. The reward does not reach back and identify which step was the good one. What it does is raise the probability of every token in a trajectory that succeeded and lower it in every trajectory that failed. Any habit that shows up more often in successful chains than unsuccessful ones will accumulate weight, slowly, across thousands of problems, purely by correlation. Backtracking is not taught. It is filtered for.
Figure 2. Outcome supervision: a tick at the bottom of the page.
A pinch of math: judging an answer by the company it keeps
The classical machinery for this — PPO , inherited from robotics — carries a second network, a critic, whose job is to predict how much reward a partial trajectory is likely to earn, so that the actual reward can be compared against expectation. A reward of one is only good news if you were expecting less than one. But the critic is roughly the size of the model itself, doubling the memory of training, and predicting the value of a half-finished chain of thought is a hard problem in its own right.
Group Relative Policy Optimization, the method behind most current reasoning models, removes it with a trick whose simplicity is the whole of it: instead of predicting what a good score would have been, just sample several answers to the same question and use their average as the yardstick.
Take a question $q$, sample a group of $G$ answers from the current model — eight, or sixteen, or sixty-four — and let the verifier score each one, giving rewards $r_1 \ldots r_G$. Then each answer’s advantage is simply how it did relative to its peers:
$$\hat{A}_i = \frac{r_i - \text{mean}(r_1 \ldots r_G)}{\text{std}(r_1 \ldots r_G)}$$If eleven of sixteen attempts failed and this one succeeded, its advantage is strongly positive and every token in it gets pushed up. If it was one of the eleven, everything in it gets pushed down. The group supplies its own baseline, and the critic network is simply gone.
The update then nudges the model to make high-advantage trajectories more likely, with one safeguard. For each token, form the ratio between the probability the current model assigns it and the probability the model assigned it when the sample was drawn:
$$\rho_{i,t} = \frac{\pi_\theta(o_{i,t})}{\pi_{\theta_{\text{old}}}(o_{i,t})}$$and clip that ratio to a narrow band around one before multiplying it by the advantage. You do not need to trace the algebra; the purpose of the clip is a single conservative instinct. These samples were drawn from a slightly older version of the model, so they are only evidence about that version. Once the current model has moved far enough that a token’s probability has changed by more than a small percentage, the clip stops the update from pushing further on the strength of stale evidence.
One consequence of the group baseline is worth noticing, because it shapes how these systems are run in practice. If all $G$ answers receive the same reward — every one correct, or every one wrong — then every advantage is zero and the question contributes precisely nothing to the update. Problems the model always solves are wasted compute, and so are problems it never solves. The useful training signal lives entirely in the band of difficulty where the model succeeds sometimes, which is why serious RLVR pipelines spend real effort filtering and re-sorting their problem sets as the model improves. The curriculum is not a nicety here; it is where the gradient comes from.
Figure 3. Signal lives only where the model succeeds sometimes.
Why the search works at all, and what it may not do
Step back and notice the assumption underneath everything above. The model can only be rewarded for a correct answer it actually produced. If it never once stumbles onto the right answer for a class of problem, the reward is always zero, the advantages are always zero, and no amount of training will teach it anything. RLVR cannot conjure a capability from nothing. It can only find, amplify, and make reliable the things the base model was already capable of doing occasionally, by luck, in one sample out of a hundred.
That framing has led to a real and unsettled argument about what these methods accomplish. There is evidence that if you let a base model take a hundred attempts at a problem and count it solved if any attempt succeeds, the base model and its RLVR-trained descendant are much closer than their single-attempt scores suggest — and, at large enough numbers of attempts, the base model sometimes covers problems its trained descendant no longer reaches. On that reading, training has concentrated probability mass onto the reasoning paths that tend to work, converting a broad and unreliable capability into a narrow and dependable one, while quietly pruning some of the breadth. That is a substantial and valuable thing to have done, and it is not obviously the same thing as having learned to reason better.
The counter-argument is that the distinction matters less than it sounds, since a capability you can only access one time in a hundred is not really a capability, and that the gains at longer training runs and harder problem sets do not look purely like sharpening. The honest position today is that both effects are present and their relative size is unresolved. It is worth holding the question open, because a great deal of confident writing about reasoning models assumes an answer to it.
Gaming a verifier that cannot be flattered
The reward is incorruptible. The channel around it is not, and this is where practitioners spend most of their time.
The verifier only checks what it was written to check. Give a coding problem a weak test suite and the model will discover the function that passes it without solving the problem — special-casing the tested inputs, or finding the degenerate return value that happens to satisfy three assertions out of three. Extract the final answer with a fragile pattern match and the model will learn to produce output that trips the pattern favourably rather than output that is correct. Leave the answer anywhere in the context and it will be found and reproduced without any intervening work; contamination between a training set and a benchmark does the same thing at a larger scale, and is the most common way an impressive number turns out to mean nothing.
Then there is everything the verifier is indifferent to. R1-Zero, optimized purely for correctness and format, produced chains of thought that switched between languages mid-sentence and were frequently unreadable — not a malfunction, simply the absence of any reward for readability, so none was provided. The eventual R1 added a cold start of supervised examples and a language-consistency term to get a model whose reasoning a person could follow. There is also a persistent pull toward length, since more tokens usually means more chances to recover from an error, and nothing in a correctness reward objects to a model that takes eight thousand tokens to answer something simple.
The pattern in all of these is the same. The reward is a perfectly faithful measurement of exactly what it measures, and everything you failed to put in it is fair game. Which is the same lesson as reward hacking in preference learning, arriving from the opposite direction: there, the measurement was unfaithful; here, it is faithful but narrow.
Where it stops
The requirement is a checker, and most of what we ask models to do does not have one. There is no program that decides whether an essay is well argued, whether an explanation was pitched correctly for its reader, whether a piece of advice was wise, whether a summary kept the right things. For that whole enormous territory, human preferences remain the only instrument we have — whether through a learned reward model, or through direct preference optimization , which dispenses with the reward model but not with the underlying reliance on a proxy for taste. RLVR did not replace preference learning; it carved out the region where something better is available, and the two now run side by side in essentially every frontier training pipeline — verifiable rewards for the domains that admit them, preference learning for taste, safety, and tone.
There is steady pressure on that boundary. Process rewards score the individual steps of a derivation rather than only its conclusion, which gives a much denser signal but requires either expensive human labelling of steps or an automatic labeller that is itself a model — reintroducing exactly the proxy we were trying to escape. Rubric-based grading and LLM judges extend the reach into softer domains at the same cost. And in the other direction, work on formalization aims to widen the verifiable region itself, translating informal problems into forms a prover can check. The interesting question for the next few years is how much of human judgement can be turned into something a machine can check without the checker becoming just another opinion.
Finally, a skill acquired this expensively does not have to be re-acquired. Once a large model has been trained with RLVR, its reasoning traces become supervised data, and a much smaller model trained on them inherits a large fraction of the ability without ever running the search — the reasoning-specific case of distillation , and the reason small open models can now do arithmetic that would have been out of reach for a model of their size two years ago.
What to carry forward
RLVR replaces a learned judge with a program that checks the answer: a test suite, a proof checker, a comparison against a known result. Because that reward is a fact rather than an estimate, it cannot be gamed by the tone or shape of a response, which means it can be optimized against far harder than a preference model — and it is that permission to push hard, more than any algorithmic novelty, that produced reasoning models. Supervision falls only on the outcome, never on the reasoning, so behaviours like backtracking and self-checking are not taught but filtered for, accumulating because they appear more often in trajectories that end correctly. The dominant method, GRPO, dispenses with the critic network of classical policy gradients by sampling a group of answers to the same question and scoring each against the group’s own average, which also means all the learning signal comes from problems the model solves sometimes — making curriculum difficulty a first-class concern. Two honest caveats travel with all of this: RLVR can only amplify capabilities the base model already exhibits occasionally, and there is real evidence that a meaningful share of its gains is sharpening a distribution rather than extending it; and the reward, while incorruptible, is narrow, so weak tests, leaky answer extraction, benchmark contamination, and unreadable or bloated reasoning are the standing failure modes. It applies only where a checker exists, which is why it now sits alongside preference learning rather than replacing it, and why the live frontier is the question of how much more of human judgement can be made checkable.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
