Overdetermined measurements

Why take more measurements than unknowns when the extra ones can only contradict each other?

Three unknowns need three equations. That is the first fact anyone learns about simultaneous equations, and it has the pleasing shape of an exact accounting: one constraint per degree of freedom, no waste.

Yet a surveyor establishing three station coordinates will take a dozen sightings; a satellite receiver solving for four unknowns will listen to nine spacecraft. Given real instruments, those extra observations are guaranteed not to agree with the first three. So the assumption worth pressing on is this: are the surplus equations a nuisance to be reconciled, or are they doing work the first three cannot?

Start with the smallest honest case. You want one number — the length of a bar. You measure once and get 100.02 mm.

What do you now know about your error? Nothing at all. You have one reading and one unknown; the reading is the estimate, exactly, with no leftover. Notice this is not a limitation of the instrument but of the arithmetic: the equation was satisfied perfectly because there was exactly enough of it to satisfy.

Measure again: 100.05 mm. The two disagree by 0.03 mm, and that disagreement is the first genuine piece of information about your measuring process you have obtained. It could not have appeared before, because before there was nothing for a measurement to contradict.

So ask what the disagreement buys. Two things, and it is worth keeping them separate, because they are usually conflated.

The first is precision. If each reading carries independent random error of size sigma, the average of m readings carries error sigma divided by the square root of m. Four readings halve it; a hundred readings cut it by ten. The errors are not correlated with one another, so they partly cancel, while the quantity itself does not.

The second is diagnosis, and it is the one that gets forgotten. With m readings and n unknowns you have m minus n leftover quantities — the residuals, the amounts by which the best-fitting answer still fails to satisfy each equation. Their size estimates how noisy the process is. Their pattern tests whether your model of the situation was right at all. And an individual residual far larger than its fellows points a finger at one specific observation as a blunder: a transposed digit, a misidentified target, a receiver hearing a reflected signal.

Now: with m equal to n, every residual is zero by construction. The fit is perfect. Ask yourself what a perfect fit tells you about whether the measurements were any good — and the uncomfortable answer is that it tells you nothing whatever, because it would have been perfect regardless. Redundancy is the only channel through which error can become visible. Remove it and errors do not go away; they go quiet, absorbed silently into the answer.

Which leaves the question of what to do with contradictory equations. You cannot satisfy them all. So do not try — ask instead for the single set of unknowns that comes closest to satisfying all of them at once, with "closest" meaning the smallest total of squared residuals. That is least squares, published by Legendre in 1805 and used by Gauss from the mid-1790s, most famously to recover the orbit of the asteroid Ceres in 1801 from a short arc of noisy sightings.

Why squares rather than, say, the sum of absolute errors? Two honest reasons and one caveat. Squaring makes the problem linear and solvable in closed form — the answer is the projection of the observation vector onto the space of achievable predictions, with the residual sticking out at right angles to it. And if the errors are independent and normally distributed, the least-squares answer is the most likely one. The caveat: squaring also means one wild outlier can drag the estimate a long way, which is why robust alternatives exist for data where blunders are expected rather than rare.

The word "overdetermined" invites the wrong picture, of a system straining under too many demands. Better to read it as over-observed: the observations exceed the minimum, and the surplus is the measurement budget's most useful part.

Two refinements are worth keeping. First, redundancy is not simply a count. Geodesists speak of a redundancy number per observation, because an observation may be surplus overall while being the only thing constraining one particular unknown — and a blunder there is undetectable no matter how many other readings exist elsewhere. Where the extra observations point matters as much as how many there are.

Second, the m minus n leftover quantities are precisely the degrees of freedom in the resulting variance estimate. Setting m equal to n does not merely leave you without a check; it leaves the noise level formally undefined, dividing by zero. The contradiction is not the price of redundancy. It is the product.

• Weighted least squares, where observations of unequal quality are trusted in proportion to their precision.
• Reliability theory in geodesy: how large a blunder must be before a given network can detect it.
• Robust estimators such as least absolute deviations, for data where gross errors are expected rather than rare.

Every term the collection defines is gathered in the glossary.