A model that turns down a request appears to be exercising judgement, and the wording encourages that reading, since it tends to say it cannot help with that, as though something in it were unable rather than unwilling. Nothing in the system is unable. The material that would answer the question sits where it sat before anyone trained the model to decline, and what was added on top is a behaviour — a learned tendency to produce one kind of continuation rather than another, installed by the same optimizer as everything else and heir to the same failures. Reading refusal as a capability removed rather than a behaviour added is the most common misunderstanding here, and nearly every surprising result follows from getting it the right way round. This piece is about how refusal is actually trained, why its two failure modes trade against each other, and why the behaviour is thinner than it looks.
Pretraining produces no reluctance
The constraint underneath all of this is worth stating precisely, because it is usually stated loosely. A base model is trained to predict the continuation of human text, so whatever is present in that text is something it was rewarded for reproducing accurately. Instructions for harm are there, along with fraud, abuse, and every technical detail anybody wrote down. The objective contains no term separating material someone would want reproduced from material nobody would: it is one number measuring how surprised the model was, and surprise has no ethics.
The base model does not decline because it has no notion of a request. It has a notion of a continuation. Given something shaped like a question it produces something shaped like an answer, not because it decided to help but because in the text it learned from, questions are followed by answers; given a manual it continues the manual. The option of not continuing never arises. Refusal is not a property that survives pretraining and then degrades under pressure. It was never there.
Refusal is trained by the machinery that trains everything else
What gets added afterwards uses no new mechanism, and that is the most useful single fact for an engineer to hold.
It arrives in two stages, neither of them special. During supervised fine-tuning the model is shown demonstrations of declining — prompts the operator does not want answered, paired with responses that decline them — under the same loss as the demonstrations teaching it to summarize a document. Then comes preference training, where two responses to one prompt are ranked and the decline is preferred. Those rankings are either compressed into a reward model and optimized against with PPO , or handed to DPO directly. Either way the machinery is indifferent to what the preference is about. A pair whose chosen response declines to help with a weapon is processed identically to one whose chosen response merely has tidier formatting.
Because the mechanism is ordinary, the pathologies are ordinary too. Every failure mode of preference learning applies here unmodified. The data is annotated by people whose thresholds disagree, so the target is noisy before any optimization begins. Methods that optimize a margin between chosen and rejected can lower the probability of both, and the displaced mass has been seen landing on the wrong side: a preferred refusal made less likely, a compliance more so. Reward models can be satisfied by the surface features of a decline rather than the decision behind it, which is one reason production refusals all sound alike; a register is easier to reproduce than a judgement. Refusal is not a safety module bolted to the side of a language model. It is one more preference, fitted like the rest.
Figure 1. Where the decline gets installed.
Two errors, one dial, and no honest way to measure either
A classifier that never errs does not exist, and refusal is a classifier whatever else it may be. So training does not move toward correctness. It moves along a curve, and both ends are bad.
At one end the model declines what it should not — over-refusal. A clinician asking about lethal doses is asking a question the job requires. A novelist asking for a violent scene is asking for fiction, which has always contained violence. A defender asking how an attack works is asking the question that precedes fixing it. All three get refused regularly, and the cause is not prudishness but the shape of the signal: they share surface features with the requests the model was taught to decline, while the context that would separate them — who is asking, and why — is absent or unverifiable.
At the other end it complies with what it should not — under-refusal — which needs no illustration.
The two rates are not independently adjustable. There is roughly one dial, and turning it down to rescue the clinician turns it up for everybody else. Where to leave it depends on the users, the deployment, and what the operator will answer for — a product decision, taken by engineers, habitually described as a technical one.
Worse, the position cannot be located empirically. Refusal benchmarks have a shorter useful life than most: publishing prompts a model ought to decline puts them where the next training run will scrape them, after which the score reports recall of specific items rather than the behaviour they sampled. The quantity scored is a judgement rather than a fact, so scoring means another model’s opinion about whether a response was a refusal and whether it should have been. And the base rate is decisive: harmful requests are a small fraction of real traffic, so a benchmark balanced between harmful and benign resembles nothing in deployment, and a false-positive rate that looks tolerable there can dominate ordinary users’ experience. These are the standing problems of evaluation in an unfavourable configuration.
A pinch of math: the overlap sets the floor
Reduce the arrangement to its simplest honest form. Suppose we have a scalar $s$ computed from a request, standing in for however the model represents the question of whether to comply, and a threshold $\tau$ above which it declines. Harmful requests are drawn from one distribution over $s$ and benign requests from another, and the two overlap. The error rates are then integrals over opposite tails:
$$\text{under-refusal} = \int_{-\infty}^{\tau} p_H(s)\, ds, \qquad \text{over-refusal} = \int_{\tau}^{\infty} p_B(s)\, ds$$Moving $\tau$ moves both, in opposite directions, and no value of it makes both small while the distributions overlap. There is one number to set and two quantities that depend on it.
What the picture makes visible is what training is actually for. Turning $\tau$ is free and buys nothing. The quantity deciding whether any threshold can work is the overlap, and the minimum total error is fixed by it alone — the Bayes error of the problem, which no decision rule beats. So the work that matters is separating the distributions: giving the model whatever it needs to place the clinician’s question and an identical-sounding one from someone else in different regions of $s$. Where two requests are indistinguishable from their text alone, the overlap is irreducible, and so is the error.
Figure 2. One threshold, two tails, and the overlap.
The behaviour is thin, and jailbreaks work on the thinness
One line of interpretability work bears on how deep the installed behaviour goes. In open-weight models, refusal appears to be mediated by a comparatively simple internal signal — close to a single direction in the residual stream , the running vector each layer reads from and writes to. That direction can be identified and suppressed, and when it is the model stops declining while ordinary task performance is largely unchanged. The finding matters for what it implies rather than for what it enables. Refusal was a thin behavioural layer over a model otherwise left alone. The knowledge is still in there. Training added a disposition rather than removing an ability. This is one line of evidence, not a settled account — a single clean direction is probably too tidy, given how features in these models share directions in the way the study of superposition describes — but it points consistently.
That thinness is the background against which refusal fails under pressure, and the account of the failures is structural. Three causes, each a class rather than a technique.
Distribution shift. Refusal was fitted on inputs resembling the safety training data, and like any learned behaviour generalizes worse the further an input sits from those. Anything that makes a request unlike those examples moves it where the behaviour was never trained, and what the model does there is whatever its general tendencies produce. This is not a special weakness of safety data. It is the ordinary generalization gap, on a behaviour where gaps are expensive.
Competing objectives. The same post-training that installs refusal also installs helpfulness, instruction-following, and a pull toward completing whatever pattern has been started. Those are separate pressures on one set of weights, and can be put into tension, so that declining requires the model to violate something else it was trained to do. The model is not being talked out of its training. It is being placed where two parts of its training disagree, and something gives.
The shared channel. Instructions and data arrive as one undifferentiated stream of tokens. Nothing in the input marks a passage as material to be processed rather than direction to be followed, and nothing in the architecture could enforce such a mark. Content the model was asked to summarize can therefore be read as content addressing it. This is the fact underneath prompt injection, and the reason injection is unfixable rather than merely unfixed: there is no parser to patch. The exposure grows with anything that widens the input: retrieved context, tool output, other agents.
None of these closes. Each is answered by training on the cases that got through, which narrows that region and shifts the frontier elsewhere, against an adversary who adapts faster than a training run completes. Refusal is a moving equilibrium maintained by continuous effort, not a property established once, and underneath it the model’s capabilities are exactly what they were.
Figure 3. A thin layer over untouched capability.
What to carry forward
Nothing in pretraining knows how to decline, because predicting the continuation of human text carries no notion of a request that should go unanswered; refusal is installed afterwards, by demonstrations during supervised fine-tuning and then by preference pairs marking a decline over a compliance, run through a reward model and reinforcement learning or through direct preference optimization. No new mechanism appears in that, which is why every pathology of preference learning is a pathology of refusal: noisy human thresholds, margin objectives that displace probability onto responses nobody chose, reward models satisfied by the register of a decline rather than the decision behind it. Refusal has two errors that trade — over-refusal, which turns away the clinician, the novelist, and the defender for resembling requests in the training data, and under-refusal, which needs no illustration — governed by roughly one dial whose setting is a product decision in technical language. Formally the two rates are tails of overlapping distributions either side of one threshold and cannot be set independently, and the floor on total error is fixed by the overlap rather than the threshold, so the only work that helps is whatever separates them. Measurement is the weakest link: refusal benchmarks become training data on publication, the thing scored is a judgement usually another model’s, and real traffic contains too few harmful requests for a balanced benchmark to describe it. And the layer is thin: in open-weight models the behaviour appears to ride on a compact internal signal that can be suppressed with little loss of capability, which fits the three structural reasons jailbreaks work — distribution shift, objectives in tension, and a token stream that cannot distinguish instruction from data. What is underneath has not been altered, and the layer above it holds only as long as it is maintained.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
