Model quantization

Why does a model keep working after most of the precision in its numbers is discarded?

A weight trained as a 32-bit float can take one of about four billion distinct values. Replace it with a 4-bit integer — sixteen values — and the model still answers questions, still writes code, still degrades only slightly on most tests. The reflex explanation is that neural networks are redundant, so they tolerate noise.

That is true, and it explains almost nothing. If tolerance to noise were the operative variable, every scheme that removed the same number of bits would do about equally well. It does not: some 8-bit schemes wreck a large language model that a different 4-bit scheme leaves nearly intact. So the number of bits is not what decides the outcome. What is?

Start with what a weight actually does. It is never used alone. Every value a layer emits is a sum of hundreds or thousands of weight-times-activation products. So the question is not how badly one weight is corrupted, but how badly the sum is.

Round each weight to the nearest representable level and you introduce an error that is small, and — if the level spacing suits the distribution — has no systematic sign. Now add many such errors together. The signal grows in proportion to the number of terms; independent errors with no preferred sign grow only in proportion to its square root. A wide layer is therefore cheaper to quantize than a narrow one, not despite its width but because of it. The cancellation is doing the work.

Notice what that argument depends on. It needs the errors to be roughly independent and roughly unbiased. Break either and it collapses — which is exactly why the better quantization methods do not simply round. Some round one weight at a time and push the resulting error into the weights not yet rounded, so the errors cancel by construction rather than by luck. Others weight the rounding decision by how large the activations that weight will multiply typically are, protecting the few that matter and spending the error on the many that do not.

Now a second question, and it is the one that separates the schemes. What do the bits in a number format buy? Two different things. A 32-bit float spends one bit on sign, eight on the exponent, and twenty-three on the mantissa. The exponent sets the range of magnitudes the format can reach; the mantissa sets the resolution within a magnitude. Half precision, FP16, splits its sixteen bits as one, five and ten. Brain float, BF16, splits them as one, eight and seven — fewer significant digits than FP16, and a far wider range. BF16 is the one that trains reliably.

That is the tell. If coarse steps were the problem, the format with more mantissa bits would win. It does not. What breaks a network is values falling outside the representable window and flattening to zero or blowing up to infinity — a range failure, not a resolution failure.

Which points at the real failure mode in large models. In a big transformer, a small number of activation channels carry values far larger than the rest, and consistently the same channels. Choose one scale for the whole tensor and you must size it to hold the outlier; everything else then collapses into the few levels nearest zero, and the layer's ordinary information is destroyed. Adding bits barely helps, because the problem is the ratio between the largest value and the typical one. Using finer-grained scales does help — one per output channel, or one per small group of weights — as does keeping the outlier dimensions in high precision, or rotating the coordinate system so that no single channel carries the excess.

This also explains the asymmetry everyone notices in practice: weights quantize far more easily than activations. Weights are fixed after training, so you can inspect their distribution and choose scales offline. Activations depend on the input, so their outliers arrive unannounced.

The word "redundancy" is doing something subtler than it sounds. What is redundant is not the weights — prune too many and the model dies — but the precision with which each weight needs to be specified. The trained solution sits in a region of parameter space where nearby settings behave similarly, so a nudge is survivable while a deletion often is not. Why trained networks land in such regions is itself argued about; the "flat minimum" account is popular and plausible, but the relationship between flatness and robustness is still contested, and I would not present it as settled.

Two further honesties. First, the damage is real, and it is unevenly distributed: aggregate benchmark scores can move very little while performance on rare facts, long contexts and multi-step arithmetic moves quite a lot, so an average is a poor instrument for detecting it. Second, the practical question everyone actually faces — for a fixed memory budget, is a large model at four bits better than a smaller one at eight? — is an open empirical question whose answer depends on the model family, the task, and the quantization method, and published results genuinely disagree.

• How error-compensating rounding decides which weights to sacrifice, and why the criterion is not weight magnitude.
• Why activation outliers appear in large transformers at all, and whether they serve a function.

Every term the collection defines is gathered in the glossary.