Error cancellation in averages
A Socratic walk-through of error cancellation in averages — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why can the average of many rough guesses beat nearly every guess that went into it?
At a country fair in 1906, Francis Galton collected 787 written guesses of the dressed weight of an ox. Almost nobody was right. Yet the middle of that pile of guesses came out at 1,207 pounds against a true weight of 1,198 — closer than all but a handful of the individual guessers. That is uncomfortable. Where did the accuracy come from, if it was in nobody's head?
Reasoning it through
REASONING #Write down what a single guess is. It is the truth plus an error: guess = truth + error. That is not a theory, it is a definition of the error term. Now what happens when you add up many guesses and divide? The truths add up and divide back to the truth exactly. The errors add up and divide by the same number.
So the whole question becomes: what does a sum of errors do? Ask what it would take for the sum to stay large. Every error would have to point the same way — all too high, or all too low. If instead some guessers overshoot and others undershoot, the sum partially collapses on itself. Some of the too-high cancels some of the too-low.
But be careful about how much it collapses. If the errors cancelled perfectly, the average would be exact, which it plainly is not. They cancel partially, and the arithmetic of that partial cancellation is worth following. When errors are independent, it is not their sizes that add but their squares — variances add, not standard deviations. So n independent errors of typical size s give a total whose typical size is s x sqrt(n), not s x n. Divide by n and the average's typical error is s / sqrt(n).
Sit with that formula for a moment, because it is the whole engine. The error shrinks — but as the square root, which is slow. Four times the sample halves the error. A hundred times the sample gives you a tenfold improvement. Accuracy is expensive, and it gets more expensive as you go.
Now the question that decides everything: what did we assume? Independence. We assumed one person's error tells you nothing about the next person's. Suppose that fails. Suppose the ox has been standing awkwardly and everyone at the fair views it from an angle that makes it look small. Now every error carries a shared component pointing the same way, and that component does not shrink with n at all. Averaging a million such guesses gives you a very precise estimate of the wrong number.
This is the difference between random error and systematic error, or bias. Random error is what averaging destroys. Bias is what averaging preserves perfectly. And the two behave so differently that more data can make things worse rather than better — because a large sample buys you confidence, and confidence in a biased estimate is precisely the dangerous state.
The clearest demonstration is the 1936 American presidential election. The Literary Digest polled from car registration and telephone lists and received over two million replies, an enormous sample by any standard, and predicted Landon would win comfortably. Roosevelt took about 61% of the vote. Gallup, sampling a few tens of thousands with attention to who was being asked, called it correctly. The Digest's error was not noise — it was that its list systematically over-represented people who could afford cars and telephones in a depression, and its low response rate compounded the skew. Two million did not help. Two hundred million would not have helped.
The analogy
THE ANALOGY #Think of a set of bathroom scales. Step on one repeatedly and the reading wobbles a pound either way; take twenty readings, average them, and the wobble largely disappears. But if the scale reads three pounds heavy because its zero is off, then twenty readings average to exactly three pounds heavy, and two thousand readings average to exactly three pounds heavy. Repetition cures the wobble and is entirely blind to the offset.
A scale's offset is fixed and could in principle be measured against a known weight, whereas the bias in a survey or a crowd is usually invisible from inside the data — there is no calibration weight to check the poll against, which is exactly why bias is so much harder to handle than noise.
Clarifying the model
THE MODEL #Three refinements, connecting the pieces.
The independence requirement is stronger than people expect, and it is about the errors, not the people. Guessers who talk to each other, forecasters reading the same wire report, or sensors sharing a miscalibrated reference all produce correlated errors. Correlation does not have to be total to hurt: even mild positive correlation puts a floor under the average's error that no increase in n can push through.
Second, the 1/sqrt(n) rule is about the standard error of the mean — the spread of the average — not about the spread of the underlying guesses, which stays exactly where it was. Bigger samples do not make individuals more accurate. They make the summary of individuals more accurate.
Third, and honestly: Galton's ox is the flattering case, and it should not be generalised too far. It works when errors sit roughly symmetrically around the truth. Crowds asked questions where a shared misconception pulls everyone the same way — how many calories in a salad, how long a software project will take — average to a confidently wrong number. The wisdom of crowds is a statement about error structure, not about crowds.
A picture of it
THE PICTURE #How to readMove right for more observations and up for errors that are independent rather than shared. Only the top-right quadrant is accurate, and note that you reach it by moving up, not merely right. Compare the two rightmost points: the 1936 Digest poll had far more data than the careful poll above it and landed in the worst quadrant, because sample size moves a point sideways and never lifts it. The bathroom scale sits at the very bottom for the same reason — repetition cannot climb the vertical axis.
What became clearer
WHAT CLEARED #Averaging works because independent errors partially cancel, and the arithmetic of that cancellation is the square root: variances add, so the error of the mean falls as 1/sqrt(n) — quadruple the sample to halve the error. But every bit of that depends on independence. Systematic bias does not cancel at all, at any sample size, which is why an unbiased poll of a thousand beats a biased poll of a million, and why the first question to ask of any large dataset is not how large it is but who or what is missing from it.
Where to go next
ONWARD #- How survey weighting attempts to repair a known bias, and why it cannot repair an unknown one.
- Why forecast ensembles gain less than 1/sqrt(n) predicts once the models share training data.
Key terms
TERMS #| Term | What it means |
|---|---|
| Standard error of the mean | the typical spread of a sample average around the true value, equal to the population standard deviation divided by sqrt(n). |
| Random error | variation with no consistent direction, which averaging reduces. |
| Systematic error (bias) | a consistent offset in one direction, which averaging leaves untouched. |
| Independence | the condition that one observation's error carries no information about another's. |
| Selection bias | error arising because the sample is drawn from a group unlike the population of interest. |
Every term the collection defines is gathered in the glossary.