The shape of a forward pass is easy enough to state. A token comes in, the tower of blocks works on it, and out the far end comes a single arrow — a few thousand numbers holding everything the model has assembled about what should happen next. But a vector is not a word, and somewhere between that arrow and the token that finally appears on your screen there is a step that is easy to wave past. That step produces one number for every entry in the vocabulary, and those numbers are the logits. They are the last thing the model computes for itself, before any of it is turned into something a person would recognize as a prediction. This piece is about where they come from, what their name means, and the one property of them that trips up almost everyone who meets them.
The problem with asking a network for odds directly
It would be tidier if the model simply produced probabilities. Why not have the last layer output, for each of a hundred thousand vocabulary entries, the chance that it comes next, and be done with it?
Because a layer cannot make the necessary promises. What a layer does is multiply by a matrix and add a bias, and the result of that is an ordinary real number — it may be negative, it may be enormous, and there is no arrangement of weights that guarantees otherwise for every possible input. Probabilities have two obligations that are hard constraints rather than tendencies: none of them may be below zero, and all of them together must total exactly one. A hundred thousand numbers produced independently by matrix multiplication will satisfy neither. You could try to force it — squash each output into the range from zero to one and then divide by the total — but the squashing saturates, and a model whose scores have all been flattened against a ceiling has no way to express that one candidate is not merely likely but overwhelming.
There is a second reason, less obvious and more important. Think about what the tower actually does on the way up. Each block reads the running vector, decides something, and adds its contribution back. Evidence accumulates by addition: one attention head notices the sentence is about cooking and pushes a little toward food words; a later feed-forward layer recognizes a particular idiom and pushes harder toward one of them. That is the natural arithmetic of the architecture. But evidence does not combine additively on the scale of probabilities. Two independent pieces of evidence, each doubling the odds of something, multiply — and multiplication becomes addition only after you take a logarithm.
So the sensible thing is to let the network work in whatever units make its own arithmetic natural, let those numbers run unbounded in both directions, and convert to probabilities in one final step at the very end. The unbounded scores are the logits. As we are about to see, they turn out to live on precisely the logarithmic scale that makes the network’s additions correct.
Where the numbers come from: a shadow cast along a direction
The conversion from arrow to scores is a single matrix, usually called the unembedding or the language-model head, and it is worth looking at one row at a time rather than all at once.
Each row of that matrix is a vector the same size as the model’s internal arrow, and there is one row per vocabulary entry. So every word the model knows owns a direction in the same space the model has been working in all along. The logit for that word is the dot product of the final arrow with that word’s direction — which, geometrically, is the length of the shadow the arrow casts along it.
That single picture explains most of what follows. If the tower has assembled an arrow pointing strongly the way that " cheese" points, the shadow along " cheese" is long and its score is high. If the arrow is roughly perpendicular to " cheese" — the model has assembled something with nothing to do with cheese — the shadow is near zero. If the arrow points away, the score is negative. And because words that mean similar things sit in similar directions, an arrow that scores well for " cheese" will usually score decently for " butter" without anything in the model having arranged that on purpose. The scores of related words rise and fall together as a consequence of the geometry, not as a separate mechanism.
Notice that this is the same trick as embedding , run backwards. On the way in, a token was converted into a vector; on the way out, a vector is compared against every token’s vector to see which it most resembles. Many models make the symmetry literal by weight tying — using one matrix for both, with the input embedding and the output direction for a given token being the same set of numbers. This saves an enormous quantity of parameters and generally costs nothing in quality, though the largest recent models often keep the two separate, having decided they can afford the memory and would rather let the two roles specialize.
Enormous is the right word, and it is worth pausing on the size. The unembedding has one row per vocabulary entry and one column per dimension of the arrow: with a vocabulary of a hundred and twenty thousand and an arrow of four thousand numbers, that is nearly half a billion parameters in a single matrix, frequently the largest in the entire model. Computing the logits means half a billion multiplications for every token generated, and the resulting score vector — a hundred and twenty thousand numbers — is itself larger than any internal activation in the network. This is why training implementations go to some lengths to avoid ever holding the full logit tensor for a long sequence in memory at once, and why the final layer is a real engineering concern rather than an afterthought.
Figure 1. A logit is a shadow cast along a word’s direction.
A pinch of math: from scores to odds, and back
Write $h$ for the final arrow the tower produces and $w_i$ for the direction belonging to vocabulary entry $i$. The logit for that entry is the plainest thing here:
$$z_i = h \cdot w_i$$A single dot product, repeated once for every word the model knows. To turn the resulting scores into odds, exponentiate each and divide by the total:
$$p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$Exponentiating does two jobs at once. It forces every number positive, disposing of the awkwardness that a score of minus four cannot be a probability, and it magnifies gaps, so that a score two points above its neighbour ends up not twice as likely but about seven times as likely.
Now run the argument backwards, which is where the name comes from. Take two entries and ask about the ratio of their probabilities. The denominators are identical and cancel, leaving $p_i / p_j = e^{z_i - z_j}$, and taking a logarithm gives:
$$\log \frac{p_i}{p_j} = z_i - z_j$$This is worth reading twice. The difference between two logits is exactly the log of the odds between the two outcomes. A logit is a log-odds — that is what the word means, borrowed from statistics, where the logit function $\log\!\big(p/(1-p)\big)$ has converted probabilities into unbounded scores since long before anyone stacked a transformer. It also settles the question left open in the previous section: the logarithmic scale on which the network’s additions are the right operation is the scale the logits already live on. When a layer adds two points to a token’s score, it has multiplied that token’s odds by about seven, and it has done so by simple addition, which is the only thing the architecture is good at.
And notice what the identity does not contain. Add the same constant to every logit and each $z_i - z_j$ is unchanged, so every probability is unchanged; in the softmax, the constant factors out of numerator and denominator and cancels. The model’s output is completely blind to where the scores sit on the number line. You do not need to keep the algebra — the intuition is that logits are a scale of evidence with no fixed zero, like altitude measured with no agreed sea level.
Only differences are real
That last observation deserves its own section, because it is the source of nearly every confusion about logits and also the reason they are useful.
A single logit, quoted on its own, means nothing at all. If someone tells you the model gave " cheese" a logit of 14.2, the correct response is that you cannot do anything with that. Fourteen point two is high if the runner-up was at 9, and it is nothing at all if forty other tokens are at 15. The scale has no zero, no units, and no fixed range; it drifts between models, between layers, and between one position in a sentence and the next. Comparing a logit from one context against a logit from another is meaningless in the same way that comparing two altitudes measured from two different arbitrary baselines is meaningless.
Differences, on the other hand, are completely well defined, and they come with a natural interpretation. Since a difference is a log-odds ratio, one unit of logit difference is a factor of $e$ — a shade under three — in the relative odds of two tokens. Two units is about seven times. Five units is about a hundred and fifty. This gives a serviceable mental conversion: a gap of a fraction of a point between two candidates means the model is close to indifferent, a gap of three or four means it has largely made up its mind, and a gap of ten means the runner-up is not in the running.
This is why logit difference is the standard measuring instrument in interpretability work, and why it turns up as the metric of choice in ablation experiments. If you want to know whether a particular attention head is responsible for the model preferring one name over another, you take the difference between those two tokens’ logits, switch the head off, and see how much the difference shrinks. Using the difference rather than either raw score removes everything that is not about the comparison you care about — including all the machinery that merely makes the model confident in general, and including the arbitrary offset. Using the probability instead would fold in the influence of a hundred thousand irrelevant tokens through the denominator, and would compress exactly the region you most want resolution in, since the difference between 99.0 and 99.9 percent is one full unit of logit and looks like almost nothing on a probability scale.
Figure 2. Only differences are real — logit gaps as odds ratios.
The dials that act on the scores
Everything that shapes how a model chooses its next word acts on the logits, in the window between the dot product and the softmax. This is the one place in the whole system where a person can reach in and intervene with a single addition or division.
Temperature divides every logit by a constant before the exponential. Divide by something less than one and the gaps widen, the leader pulls away, and the model becomes decisive to the point of monotony; divide by something greater than one and the gaps close, the field bunches up, and unlikely words start winning. It is the same dial, doing the same thing, as the temperature used in distillation to soften a teacher’s output so a student can see the shape of its uncertainty — there it is applied to make the fine print visible, here to make the model more or less adventurous, but the arithmetic is identical.
Truncation acts on the ranking rather than the spacing. Keeping only the highest $k$ scores, or only as many of the top scores as are needed to account for some share of the total probability, and setting the rest aside before sampling, is a way of allowing variety without allowing absurdity — the tail of a hundred thousand tokens holds a great deal of accumulated probability mass made up entirely of words that would be ridiculous.
Then there is direct interference. Adding a fixed amount to a particular token’s logit makes it correspondingly more likely, and subtracting makes it less; this is what a logit bias setting in an interface does. The limiting case is the useful one: set a token’s logit to negative infinity and it becomes strictly impossible, since its exponential is exactly zero. Doing that to every token that would violate a grammar is how constrained decoding works — how a model can be made to emit only valid JSON, or only one of five permitted answers, with a guarantee rather than a request. The model is not being persuaded to comply. The alternatives have been removed from the ballot.
Reading the scores early: the logit lens
Here is a trick that is genuinely lovely, and it follows from nothing more than the observation that the unembedding is just a matrix you can apply whenever you like.
The tower does not change the shape of the arrow. The vector flowing through it after block four has exactly the same dimensions as the vector that comes out of block thirty-two. So there is nothing stopping you from taking the half-finished arrow at some intermediate layer and multiplying it by the unembedding anyway, to read off the logits the model would have produced had it stopped there. This is the logit lens, and what it shows is that the model’s prediction is not computed at the end but drafted early and revised continuously.
Run it layer by layer on a sentence and you can watch the answer come into focus. In the early blocks the top candidates are generic and often just reflect the current token. Somewhere in the middle the right kind of word starts to surface — the right part of speech, the right semantic neighbourhood — and in the upper blocks a specific token climbs and the field settles. Occasionally you catch a reversal, where a plausible early leader is actively suppressed by a later layer that has noticed something. It makes the residual stream legible as what it is: a running draft in the model’s own vocabulary, which every block reads, amends, and passes on.
The technique comes with caveats, and it is honest to name them. Intermediate arrows have not been through the final normalization the unembedding expects, so the readings are distorted, and various refinements exist that learn a small correction for each layer before applying the lens. Some layers are simply not writing in a form the unembedding can read, and the output is noise. But as a way of seeing that depth is refinement rather than a single enormous computation at the end, it is hard to beat.
Figure 3. The logit lens — a draft written early and revised.
Two practical notes
Exponentiating a logit of, say, ninety produces a number too large for the arithmetic to hold, and the result is an infinity that poisons everything downstream. The universal fix exploits the shift invariance we established above: subtract the largest logit from all of them before exponentiating. Every score becomes zero or negative, every exponential lands safely between zero and one, and because the shift cancels, the answer is exactly unchanged. Every softmax implementation in every framework does this silently.
Relatedly, the training loss is always computed directly from the logits and never from probabilities that were computed first. Taking the logarithm of a small probability that has already been rounded to the nearest representable number loses precision that cannot be recovered, whereas the combined operation of softmax-then-logarithm simplifies algebraically into something stable. If you have ever wondered why the loss function in a codebase is named for cross-entropy with logits, this is why: it is not being handed probabilities and it does not want them.
What to carry forward
The logits are the model’s raw scores — one per vocabulary entry, produced by taking the final arrow out of the tower and dotting it against a direction belonging to each token, so that a logit is the length of the shadow the model’s assembled state casts along the word in question. They are unbounded because the network’s arithmetic is additive and cannot promise otherwise, and the softmax converts them into odds at the very end. The name is exact rather than decorative: the difference between two logits is the logarithm of the odds between those two outcomes, which means logits are the scale on which the tower’s layer-by-layer additions correspond to multiplying odds. The single most important consequence is that only differences carry meaning. Adding a constant to every logit changes no probability at all, so a lone score quoted in isolation is uninterpretable, while a gap of one unit is reliably a factor of about three in relative odds — which is why logit difference is the standard yardstick in interpretability work and the metric of choice for ablation. Every knob that shapes generation acts here, in the gap before the softmax: temperature divides the scores, top-$k$ and top-$p$ truncate them, a bias adds to one of them, and negative infinity removes a token from consideration outright, which is how constrained decoding enforces a format rather than requesting one. And because the unembedding is only a matrix, it can be applied to a half-finished arrow partway up the tower — the logit lens — revealing the prediction as a draft that is written early and revised block by block, rather than computed in one stroke at the end.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
