Stratified sampling

Why can splitting a population into groups before sampling beat sampling the whole at random?

Random selection is the thing that makes a sample trustworthy. Give every unit an equal chance and you have removed yourself from the choosing; any structure you impose afterwards looks like interference sneaking back in.

So it is worth asking plainly why the professionals almost never do it. A national survey does not draw names from one hat. It carves the population into regions, age bands, farm sizes — and then samples inside each piece, often at deliberately unequal rates. The question is what that carving buys, given that the simple version was already unbiased.

Take a town of 1,000 homes. Nine hundred are ordinary houses worth around 200 thousand, varying by about 20 thousand either way. One hundred are large properties worth around 2 million, varying by about 200 thousand. You want the average value, and you can afford to appraise 100 homes.

Draw those 100 at random. What decides your answer? Ask how many large properties you happen to catch. On average you catch ten — but eight is unremarkable, and so is thirteen. Each one you catch or miss moves the total by roughly 2 million against a sample of 100, which is 20 thousand on the mean. The estimate swings enormously, and it swings for a reason that has nothing to do with measurement error: it is the luck of the draw between two very different kinds of thing.

Work the numbers and the standard error of that simple random estimate is about 54 thousand, on a true mean of 380.

Now make the obvious move. You already know which homes are large — that is on the valuation roll, before any sampling. So do not let chance decide the mix. Take 90 ordinary houses and 10 large properties, deliberately, and estimate each group's mean from its own sample, then combine them with their known weights.

What has changed? Ask where the variability went. The huge spread between 200 thousand and 2 million was the main thing driving the error, and you have removed it from the sampling problem entirely, because the proportions are no longer being sampled at all — they are fixed at their true values. What remains is only the variation inside each group, and each group is far more uniform than the town as a whole. The standard error drops to about 6.6 thousand: an eightfold improvement from no extra appraisals whatever.

Here is the specialization at the heart of it. Each stratum gets its own dedicated sample, its own estimate, and its own share of the effort, and each of those estimators works inside a narrow, homogeneous world it can handle well — rather than one general-purpose estimator trying to cope with a population that contains two incompatible kinds of thing.

Push the specialization one step further and ask: should each group really get effort in proportion to its size? Where is the difficulty? The large properties vary by 200 thousand, ten times as much as the ordinary houses. Effort spent on a group that is nearly uniform is wasted; effort spent where readings scatter is what actually buys precision. Neyman's answer, from 1934, is to allocate in proportion to a stratum's size multiplied by its internal spread. Here that gives about 47 ordinary houses and 53 large properties — half the budget on a tenth of the population — and the standard error falls again, to about 3.8 thousand.

Notice that this is unequal sampling on purpose, and it is still unbiased, because the known weights put each stratum back in at its true share when the estimates are combined.

Three refinements keep this honest.

Stratification helps only to the extent that the strata differ. Split the town by whether the house number is odd or even and the two groups have the same mean and the same spread; the between-group variation you removed was zero, and you have gained nothing (and lost a hair through the arithmetic of small samples). All the benefit comes from strata that are internally similar and mutually different — the more the grouping variable explains, the larger the gain.

It is not a bias correction. Simple random sampling was already unbiased; what stratification reduces is variance, meaning how far a typical sample lands from the truth. Those are different complaints and stratification answers only the second.

And it has a real cost, which is information. You need the stratifying variable for every unit in the frame in advance, plus the true stratum sizes for the weights. Neyman allocation additionally wants each stratum's internal spread, which you usually do not know and must guess from a prior survey — a good guess produces a large gain, and a poor one merely returns you towards the proportional case rather than doing harm.

• Cluster sampling, which groups units for the opposite reason: to reduce cost rather than variance, and which usually increases the error.
• Post-stratification and raking, for when the grouping variable is only observed after selection.
• Design effects, the single number surveys use to report how much a complex design beat or trailed simple random sampling.

Every term the collection defines is gathered in the glossary.