Degrees of freedom

Why does estimating a mean from your data cost you one observation's worth of information?

You measure ten things and want their spread. The obvious recipe is to take the average, measure how far each value sits from it, square those distances, and average them. Ten numbers, divide by ten. Instead every textbook tells you to divide by nine, and usually explains it with the phrase "you lose a degree of freedom" — as if one of your measurements had been confiscated at the door.

But nobody took a measurement away. All ten values are still sitting there. So what exactly is the missing thing, and in what sense was it spent?

Try a smaller case, because the arithmetic stops hiding at three. Suppose I tell you three numbers have a mean of 10, and then I tell you the first is 7 and the second is 14. What is the third? You did not need to be told — it is 9, and it could not have been anything else.

So hold that. Once the mean is fixed, the last value is not free. Two of the three could be anything at all; the third is determined. That is not a metaphor about information, it is a hard arithmetic constraint. And notice its exact form: the deviations from the mean must sum to zero. Always. Not approximately, not on average — exactly, by the definition of the mean.

Now ask what that constraint does to the quantity you actually care about. You wanted the typical distance from the true population centre, but you do not have that; you have the sample mean. And the sample mean is not an independent landmark — it was computed from these very points, so it has been dragged toward wherever they happened to fall. Ask yourself: is the sample mean therefore too close to your data, or too far from it?

Too close, necessarily. The sample mean is the single point that minimises the sum of squared distances to your data — that is what "mean" does. Any other point, including the true population mean, gives a larger total. So squared deviations measured from your own sample mean are systematically too small, every single time, never too large. The bias is not random; it is structural, and it always points the same way.

By how much? Work through the expectation and the answer comes out remarkably clean: the expected sum of squared deviations from the sample mean is (n - 1) times the true variance, not n times it. That is Bessel's correction. Dividing by n - 1 rather than n does not add a fudge factor — it exactly undoes a shortfall whose size the algebra pins down.

Here is the picture that makes it feel inevitable rather than arithmetic. Your n measurements are a single point in n-dimensional space. The constraint "deviations sum to zero" says the residual vector is perpendicular to the direction where all coordinates are equal — the line along which the whole sample shifts up or down together. Perpendicular to one direction in n dimensions leaves you a subspace of dimension n - 1. The residuals do not merely tend to live there; they cannot leave it. So they carry n - 1 independent coordinates' worth of variation, and dividing by that number is dividing by the dimension they actually occupy.

That framing generalises immediately, which is the real test of whether an idea is right. Fit a straight line and you impose two constraints — residuals sum to zero and are uncorrelated with the predictor — leaving n - 2. Fit p parameters, leave n - p. Every fitted parameter is one direction the residuals are forbidden to point in. Nothing is being consumed metaphorically: dimensions are being removed, and they are removed one per constraint.

Several refinements sharpen this, and one of them is a genuine correction.

If you happen to know the true population mean — from theory, or from a much larger reference sample — then divide by n. Nothing was spent, because no constraint was imposed by your data. The n - 1 is not a property of samples; it is a property of estimating the centre from the same data you are measuring spread in. That is the whole content of the rule.

The correction people miss: dividing by n - 1 gives an unbiased estimate of the variance, and it does not give an unbiased estimate of the standard deviation. The square root of an unbiased estimate is not itself unbiased, because the square root is a concave function — so the usual sample standard deviation still runs slightly low. Almost nobody bothers to correct this further, and for most purposes that is fine, but the common claim that n - 1 "makes the standard deviation unbiased" is simply false.

Finally, the count is not always an integer. In mixed models and in kernel smoothers the effective degrees of freedom is a trace of a projection matrix and can land at 4.7. That is not a fudge either — it is the same dimension-counting idea applied to a fit that constrains directions partially rather than completely.

• Why the t-distribution has heavier tails than the normal, and how its degrees-of-freedom parameter is the same count reappearing.
• Cochran's theorem, which makes the geometric argument here precise.
• Effective degrees of freedom in ridge regression and smoothing, where the count becomes continuous.

Every term the collection defines is gathered in the glossary.