The article on QLoRA introduced the idea of describing a number in fewer bits, and applied it to a frozen model being adapted. Training a model from scratch in reduced precision is a harder problem, because a frozen weight only has to be read accurately while a weight under training has to accumulate millions of tiny corrections without any of them being lost to rounding. The tension is sharp: half-precision arithmetic runs several times faster and halves the memory traffic, which is the difference between a training run costing one month and three, but the updates the optimizer wants to apply are frequently smaller than the smallest change a half-precision number can represent. This piece is about how that tension is resolved, why the resolution hinges on a detail of how floating-point numbers are laid out, and what changed when the format itself was redesigned for the purpose.
What a floating-point number spends its bits on
A floating-point number divides its bits into two jobs. Some encode the exponent, which sets the scale — how large or small the number is, in powers of two. The rest encode the mantissa, the significant digits within that scale. Exponent bits buy range; mantissa bits buy precision.
Single precision, the traditional default, uses eight exponent bits and twenty-three mantissa bits in thirty-two, which is generous on both counts. The original half-precision format, fp16, uses five and ten in sixteen. Halving the total was the point, but notice which half was cut hardest: three exponent bits were surrendered, and the smallest positive number the format can represent normally is about $6 \times 10^{-5}$, against roughly $10^{-38}$ for single precision. That is not a small reduction in range. It is thirty-three orders of magnitude.
This matters because gradients in a large network are small. Deep in a tower, for a parameter that barely affects the loss, a gradient of $10^{-8}$ is entirely ordinary. In fp16 that number is not small — it is zero. It underflows, silently, and the parameter receives no update at all. A training run can appear to be progressing while a substantial fraction of its gradients are being quietly discarded.
Figure 1. Where three formats spend their bits.
Three fixes, and why all three are needed
The standard recipe for training in fp16 has three components, each addressing a different failure, and leaving any one out breaks it.
A master copy in full precision. The optimizer keeps the authoritative weights in thirty-two bits and casts a half-precision copy for the forward and backward passes. The reason is accumulation. A weight of magnitude one, updated by $10^{-7}$, is unchanged in fp16 — the update is far below the gap between representable numbers at that scale, so adding it rounds straight back to where you started. Do that ten million times and the weight has still not moved, even though the corrections summed to something substantial. Keeping the master copy wide means every tiny update lands somewhere, and the accumulated total eventually becomes visible in the half-precision copy. This is why the memory arithmetic in the article on AdamW counts a four-byte master weight alongside the two-byte working weight.
Loss scaling. To stop gradients underflowing, multiply the loss by a large constant before the backward pass. Every gradient is scaled up by that same constant — the chain rule guarantees it — lifting them out of the range where fp16 rounds to zero. Divide them back down before the optimizer applies them, and the mathematics is unchanged. The scale cannot be fixed in advance, because too small a value fails to rescue the smallest gradients and too large a value overflows the largest, so implementations adjust it dynamically: raise it steadily while things are fine, and on any step where a gradient comes back as infinity or not-a-number, discard that step entirely and halve the scale. A training run in fp16 therefore throws away an occasional batch by design, which is disconcerting to see in a log and is working as intended.
Selective full precision. Some operations are performed in thirty-two bits regardless. Anything that sums many terms — the reductions inside normalization , the softmax denominator, the cross-entropy computation — accumulates rounding error in proportion to how many terms it adds, and in half precision the accumulated error can exceed the answer. The rule of thumb is that matrix multiplications are safe in low precision because the hardware accumulates them in a wider register internally, while explicit reductions in the model’s own code are not.
The format that made most of this unnecessary
The three-part recipe worked and was universally used, and it was also fragile — loss scaling in particular is a moving part that can fail in ways that look like a bad learning rate. The eventual fix was to change the number format rather than to keep working around it.
bfloat16 is also sixteen bits, but it allocates them differently: eight exponent bits and seven mantissa bits. It keeps single precision’s exponent field exactly and pays for it out of the mantissa. The result is a format with the same range as thirty-two-bit floating point and considerably less precision.
That is the correct trade for this problem, and understanding why is the point of this piece. Gradients do not need many significant digits. They are noisy estimates in the first place — the whole argument of the article on batch size is that the gradient is a sample, not a measurement — so knowing one to three decimal places rather than seven costs almost nothing. What gradients absolutely do need is to exist, and existing means being representable. Range is what they need; precision is what they can spare. The original format optimized the wrong one.
With bfloat16, gradients do not underflow, so loss scaling becomes unnecessary and disappears from the recipe entirely. The master copy remains, because the accumulation problem is about precision and bfloat16 has less of it than fp16 did. Selective full precision for reductions remains. But the fragile, dynamic, occasionally-run-destroying component is gone, and essentially all current large-model training uses bfloat16 for this reason.
Figure 2. The fp16 recipe, and which parts bfloat16 retired.
A pinch of math: why accumulation needs the wide copy
The one part worth making precise is why a master copy is required even when range is not an issue, because it is the part people most often assume they can skip.
A floating-point format with $m$ mantissa bits represents numbers near magnitude $x$ with a spacing of about $x \cdot 2^{-m}$. Adding anything smaller than half that spacing leaves the value unchanged after rounding. So the condition for an update $\Delta w$ to have any effect on a weight $w$ is:
$$|\Delta w| \;\gtrsim\; |w| \cdot 2^{-(m+1)}$$For bfloat16, $m = 7$, so an update must be roughly one part in two hundred and fifty of the weight’s own magnitude to register at all. A weight of magnitude $0.02$ therefore ignores any update below about $10^{-4}$ — and typical updates late in training are far smaller than that. In single precision, with $m = 23$, the threshold is one part in sixteen million, and the same updates land comfortably.
This is a different failure from underflow and is not fixed by having range. The number $10^{-7}$ is perfectly representable in bfloat16; it simply vanishes when added to something eight orders of magnitude larger. The master copy exists so that these vanishing updates accumulate somewhere they can be seen, and it is the reason low-precision training saves less memory than the bit count suggests — the weights are stored twice.
Figure 3. The smallest update a weight will notice.
Eight bits, and where the line currently sits
The obvious next step is eight bits, and it is now in production use on hardware that supports it, though with more care than sixteen required.
The formats offer a choice between range and precision — one with more exponent bits, one with more mantissa — and the practice is to use different formats in different places: the one favouring precision for weights and activations in the forward pass, the one favouring range for gradients in the backward pass, on the same reasoning that motivated bfloat16. Scaling becomes essential again, but per-tensor rather than global: each matrix carries its own scale factor, chosen from the recent history of its values, so that its numbers occupy the available range. Some designs go further and scale per block of a few dozen values, which handles the outlier dimensions described in the article on softmax — those enormous activations that support attention sinks would otherwise force a scale that crushes everything else in the tensor.
What is not done in eight bits is the accumulation. Matrix products are computed from eight-bit inputs but summed in a wider register, the master weights stay in thirty-two bits, and the optimizer state stays wide. The pattern is the same one running through this whole subject and through QLoRA : store and multiply narrow, accumulate wide. Where the field will stop is not clear — four-bit training has been demonstrated and is not yet routine — but the principle has held at every step so far, which suggests that whatever precision the multiplications end up using, something behind them will be keeping a careful wider tally.
What to carry forward
Training in half the bits halves memory traffic and multiplies arithmetic throughput, but a weight being trained must absorb millions of corrections far smaller than itself, which is a harder demand than merely being read accurately. A floating-point format splits its bits between an exponent, which buys range, and a mantissa, which buys precision; the original half-precision format cut the exponent hard, so ordinary small gradients underflow silently to zero. The classical recipe patched this three ways: a full-precision master copy of the weights so tiny updates can accumulate somewhere rather than rounding away; loss scaling, multiplying the loss by a large dynamically-adjusted constant to lift gradients out of the underflow region, discarding any step that overflows; and keeping explicit reductions — normalization, softmax, the loss itself — in full precision, since summing many terms accumulates error in proportion to their number. bfloat16 made most of that unnecessary by reallocating the bits: full single-precision range, much less precision, which is the right trade because gradients are noisy estimates that need few significant digits but must at least be representable. Loss scaling disappeared with it; the master copy did not, because the accumulation problem is about precision, and an update smaller than roughly one part in $2^{m}$ of a weight’s magnitude vanishes on addition no matter how well the format can represent it in isolation. Eight-bit training now works using per-tensor or per-block scales and different formats for the forward and backward passes, and it obeys the same rule as everything before it: multiply narrow, accumulate wide.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
