The machinery that makes a large model good is well enough known: attention letting every word survey every other, a tall tower of blocks, and a long, expensive training run that settles billions of knobs into a capable configuration. Capability comes at a price. The best models are enormous, and every answer burns memory and time in proportion to their size — tolerable in a demonstration, punitive across billions of requests. So the wish is obvious: keep most of what a giant model knows while paying for a small one. Distillation is the technique that grants it, by transferring the large model’s judgement into a smaller one. This piece is about how the transfer works, why it works better than the size difference suggests, and what the student inherits along with the skill.
The problem with simply training a small model
The obvious approach skips the big model entirely and trains a small one from scratch on the same data. Sometimes that is good enough. But a small model trained this way tends to plateau below what its size would allow — it has the raw capacity to be better and never gets there. The reason is subtle, and it is the key to everything that follows.
Ordinary training presents a piece of text, asks for the next word, and scores the answer against the single word that actually came next. That correct word is treated as the whole truth and every other word as equally wrong. This throws away an enormous amount of information. Take the sentence “I poured milk into my…”. The real next word might be “coffee”. Ordinary training says: “coffee” is right, and “tea”, “cereal”, “cup” and “airplane” are all simply wrong, with no distinction among them. But “tea” and “cereal” are nearly right, “cup” is plausible, and “airplane” is absurd. That texture — the relative plausibility of all the wrong answers — is real knowledge about the world. Standard training discards it.
The teacher’s real gift: the shape of its uncertainty
Here is the insight that makes distillation work. A well-trained large model does not discard that texture. Asked to predict the next word it hands back a full set of odds across every possible word — a high number on “coffee”, smaller but respectable numbers on “tea” and “cereal”, a tiny number on “airplane”. These graded odds are the soft targets, and they encode more than the answer: they encode how the model has organized the whole landscape of possibilities around it. The relative sizes of the wrong answers are the organization made legible — that tea and coffee are cousins, that airplanes have no place in a cup.
Distillation’s core move is to stop teaching the small model from the hard, one-word truth and teach it from the teacher’s soft odds instead. The lesson changes from “the answer is coffee” to a ranking over the whole vocabulary in which coffee leads, tea and cereal follow, and airplane is ruled out. The standard names make the roles plain: the big model is the teacher, the small model is the student, and the student imitates not the teacher’s final choice but the full distribution behind it.
A pinch of math: turning scores into odds
One piece of machinery makes this precise, and it is a familiar one. At the top of its tower, a model does not produce odds directly; it produces a raw score — a logit — for every possible word. Call these scores $z_1, z_2, \ldots$, one per word in the vocabulary. The softmax turns them into odds that sum to one:
$$p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$Each score is exponentiated and divided by the total, so larger scores become larger probabilities and everything adds up to one — arithmetic you could do on the back of an envelope for a vocabulary of five. Nothing unusual — this is how every model turns its final scores into a guess.
The trouble is that a confident teacher produces a very spiky softmax. If “coffee” scores far above everything else, almost all the probability lands on “coffee” and the interesting texture — the small differences between “tea”, “cereal” and “airplane” — is squashed near zero, where the student cannot see it. The whole gift of distillation lives in those small numbers. Something has to bring them out.
Temperature: softening the odds so the student can see them
The fix is a single dial, the temperature, written $T$. Every score is divided by $T$ before the softmax:
$$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$$At $T = 1$ this is the ordinary softmax. Raise $T$ above one and the scores are pushed closer together, which softens the distribution — the spike over “coffee” comes down and the small numbers over “tea” and “cereal” rise into view. Distillation runs teacher and student at the same raised temperature, so the student trains against a spread distribution rather than a hard spike. The name is borrowed from physics, where high temperature spreads a system out and low temperature freezes it into place, and the intuition carries over unchanged.
Figure 1. What temperature does to the teacher’s odds.
Measuring how well the student is imitating
The student needs a score for how closely its own soft odds match the teacher’s. The yardstick is the Kullback–Leibler divergence, written $\mathrm{KL}$, which measures how far one distribution sits from another. With the teacher’s odds $p$ and the student’s $q$:
$$\mathrm{KL}(p \parallel q) = \sum_i p_i \log \frac{p_i}{q_i}$$The divergence is zero when the two distributions are identical and grows as they drift apart, and it weights each term by $p_i$ — the teacher’s own probability. So it punishes the student hardest for disagreeing exactly where the teacher was most confident, and barely at all where the teacher had nothing to say. That direction is a choice with consequences, and the article on KL divergence sets them out: forward KL is mass-covering, so the student is pushed to reproduce the teacher’s whole distribution, wrong answers included. This term, computed on the temperature-softened odds, is the soft loss.
Two lessons at once: the combined loss
The hard truth is not usually thrown away, because the word that actually came next is still reliable information. So the student trains on two lessons blended. The first is the ordinary one: match the single correct word, as in normal training — the hard loss. The second is the distillation lesson: match the teacher’s softened odds — the soft loss just built. A mixing weight $\alpha$ between zero and one combines them:
$$L = \alpha \, L_{\text{hard}} + (1 - \alpha) \, L_{\text{soft}}$$The dial $\alpha$ decides how far the student trusts the cold, correct answer against the teacher’s richer but second-hand judgement. Lean toward the hard loss and the student behaves like an ordinary trained model; lean toward the soft loss and it tracks the teacher. Dividing scores by $T$ shrinks the gradients that drive learning by roughly a factor of $T^2$, so the soft loss is multiplied back by $T^2$ to keep the two pulling with comparable strength. It is a bookkeeping correction, not an idea. The recipe is then: run teacher and student warm, ask the student to imitate the softened odds while also getting the plain answer right, and descend on the blended loss with the same machinery any other model is trained with.
Figure 2. The two lessons that make the blended loss.
Why it works: learning the reasons, not just the answers
Step back and the reason distillation succeeds is visible. Ordinary training supplies a correct outcome and nothing else. Distillation supplies the teacher’s whole ranking — how sure it was, what near-misses it entertained, what it dismissed. The teacher’s uncertainty is its most instructive output: a compressed map of how the possibilities relate, handed over on every example at no extra cost once the teacher has run.
This is why a distilled small model routinely beats a same-sized model trained from scratch. Both have identical raw capacity. The distilled one was taught with richer lessons, so it fills that capacity with better-organized structure. The gain is in the organization, not the size.
Variations on the theme
Matching softened output odds is the classic recipe, and the family is broader. Sometimes the student matches not just the teacher’s final odds but the intermediate values flowing through the middle of the teacher’s tower, so it reproduces inner workings rather than conclusions. Sometimes there is no separate teacher and student at all: a large model generates a body of high-quality answers and a smaller model trains on that text in the ordinary way. That looser form has become the common one for building lean open models, and it is worth noticing what it gives up — training on generated text sees only the words the teacher happened to emit, not the distribution behind them, which is the entire quantity the classic recipe was built to transfer. Distillation also pairs with the other shrinking techniques, trimming unneeded knobs or storing each knob in fewer bits: distillation shrinks what the model knows into fewer knobs, those methods shrink the cost of each knob.
What the student inherits
The transfer is faithful, and that is the cost as well as the mechanism. Mass-covering KL asks the student to reproduce the teacher’s distribution rather than the truth, so a teacher’s confident errors arrive intact and are learned as carefully as its correct rankings; nothing in the loss distinguishes them. The teacher is also a ceiling. A student trained to match a distribution cannot be pushed past it by more training on the same teacher, which is why gains from distillation stop where the teacher’s competence stops. And the transferred skill is not the capability that produced it. Where a reasoning model found its habits through an expensive search against a verifier, as the article on RLVR describes, a distilled student imitates the traces and never runs the search — it arrives holding the behaviour without the means of having discovered it. The saving is real and it is a saving on inference, not on the teacher: somebody paid to train the large model, and somebody paid to run it across the whole distillation set.
Figure 3. What crosses over with the competence.
What to carry forward
Distillation transfers a large model’s judgement into a small one by training the student on the teacher’s soft odds rather than on hard, single-word answers. The teacher’s graded uncertainty — how it ranks every wrong answer, not merely which one is right — is the lesson ordinary training throws away. A temperature dial brings that texture into view, the KL divergence measures the student’s imitation, and a mixing weight $\alpha$ blends the soft lesson with the ordinary hard one, with a factor of $T^2$ restoring the gradient the softening removed. The payoff is a model carrying much of a giant’s competence at a fraction of its running cost, and it is why so many small, fast models are trained against large ones rather than from scratch. What travels with the competence is everything else in the teacher’s distribution: its errors, its ceiling, and none of the search that found the behaviour in the first place.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
