Testing many questions at once

Why does a careful study asking twenty harmless questions usually turn up one striking result?

A trial measures one treatment against one placebo, but records twenty outcomes: blood pressure, sleep, weight, mood, six blood markers, and so on. Each is analysed properly, at the conventional 5% threshold. One of them comes back significant, and it is the striking one — the drug appears to improve sleep.

Nothing in that description is misconduct. Every test was run correctly. So why be suspicious of the finding, and suspicious before knowing anything about sleep or the drug? The unease comes from the shape of the exercise rather than the substance of the claim — an odd thing for a piece of evidence to depend on.

Ask what the 5% threshold actually promises. It promises that if the treatment does nothing, a result this extreme turns up in at most one run in twenty. That is a guarantee about a single test, and about a rate rather than the particular result you hold.

Now suppose the drug is inert — every one of the twenty nulls is true. Each test independently has a 95% chance of behaving. The chance that all twenty behave is 0.95 multiplied by itself twenty times, and the chance that at least one misfires is the complement: 1 − 0.95²⁰. Work it through: 0.95²⁰ ≈ 0.3585, so the probability of at least one "significant" result is about 0.64. A useless drug produces a headline nearly two times in three.

The general form is 1 − (1 − α)^k for k independent tests at level α. At k = 1 it is α, as promised; as k grows it approaches 1, and quickly — at 100 tests, about 0.994. A simpler quantity hides here too: the expected number of false positives among k true nulls is just kα, so for twenty tests at 5%, exactly one. The study was, on average, always going to find one thing.

Notice what has gone wrong conceptually: the α chosen governs a single test, while the claim being made — "this study found something" — is about the whole family. The error rate for the claim actually made was never 5%.

What fixes it? Bound the family. If you want the probability of any false positive to be at most α overall, Boole's inequality does the work: the probability of a union is at most the sum of the probabilities, so testing each hypothesis at α/k gives a total of at most k × (α/k) = α. That is the Bonferroni correction — 0.0025 per test for our twenty — crude, valid regardless of dependence, and conservative. Under independence you can do slightly better with 1 − (1 − α)^(1/k); Holm's step-down procedure does better still with no independence assumption at all and should generally be preferred.

But should you always want that guarantee? In a scan of 20,000 genes, guarding against a single false positive drives the per-test threshold so low that real effects are lost. Benjamini and Hochberg's answer changes the target: control not the chance of any error but the expected proportion of errors among the hypotheses you declare — the false discovery rate. Accepting that 5% of your announced discoveries are wrong is often a more sensible bargain than insisting that probably none are.

Three refinements, and the first is the one people get wrong.

The 1 − (1 − α)^k formula assumes independence, and real tests rarely are. Twenty outcomes measured on the same patients are correlated, and positive correlation lowers the family-wise error rate below the formula's value — the tests tend to misfire together rather than separately. So 0.64 is the independent-case figure and, for a typical trial, an upper-ish estimate rather than a precise one. The direction of the argument survives; the exact number should not be quoted as if it were the study's.

Second, this is not the same problem as a screening test flagging mostly healthy people. That is one test and a low base rate: few positives are real because few were ill to begin with. Here the count of tests does the damage. The two multiply, which is Ioannidis's point about published findings — a field running many tests on mostly-false hypotheses produces a literature in which a large fraction of significant results are wrong, and neither factor alone explains it.

Third, and genuinely contested: whether to correct at all. Rothman argued in 1990 that adjustment penalises the thorough investigator and that the "family" of tests has no principled boundary — your twenty, or every test in your career? The counter-argument is that uncorrected exploratory findings are routinely reported as confirmatory ones. The workable compromise is structural rather than arithmetic: name one primary outcome in advance, correct within that family, report the rest as exploratory. Gelman and Loken's sharper version is that you need no explicit tests at all — flexible choices about which subgroup, which covariate, which transformation produce the same inflation with a test count of one.

How would you falsify a suspect result? Demand replication in fresh data: multiplicity produces findings that do not repeat, and a real effect does. The famous demonstration is ISIS-2, whose authors deliberately subdivided their aspirin data by astrological birth sign and reported that aspirin failed to help patients born under Gemini and Libra — a real subgroup result, at conventional significance, from an enormous trial.

• How pre-registration and registered reports remove the multiplicity by fixing the analysis before the data exist.
• Why subgroup analyses are especially prone to this, and what an interaction test does that a within-subgroup test does not.

Every term the collection defines is gathered in the glossary.