Survivorship bias
A Socratic walk-through of survivorship bias — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why did reinforcing the bullet holes on returning bombers armour exactly the wrong part of the plane?
During the Second World War, a group of statisticians at Columbia — the Statistical Research Group, with Abraham Wald among them — were asked where to add armour to bombers. Armour is heavy, so it can only go in a few places. The available evidence was a survey of aircraft that had come back, marked up with where they had been hit.
The obvious reading is that the holes show where the fire lands, so the holes are where the armour belongs. The famous punchline is that you should armour the places with no holes. Both readings sound like slogans, and neither tells you why. What is it about a survey of returning aircraft that makes a perfectly accurate record point in the wrong direction?
Reasoning it through
REASONING #Ask first what the survey actually measures. Not "where aircraft get hit". Every hole in it belongs to an aircraft that came home. So each mark records two things at once: that a hit landed there, and that the aircraft survived it. The record is a product of the thing you want and the thing you are trying to estimate.
Now separate them. Suppose fire arrives across the airframe in some pattern — not necessarily even, but a pattern set by the enemy and the geometry of the attack, not by us. For each region, the density of holes you observe among survivors is roughly the rate at which that region is hit, multiplied by the chance an aircraft survives a hit there. Look at a region with plenty of observed holes: hits there are evidently survivable. Look at a region with none: either nothing ever hits it, or nothing that does comes back.
That is the fork on which the whole inference turns, and it is where the popular version of the story quietly cheats. "Armour where the holes are not" is only sound if you already have reason to think the region is being hit at a comparable rate. If the enemy simply never engaged from an angle that exposed some panel, the absence of holes there means nothing at all, and armouring it wastes weight. So the argument requires an assumption about the distribution of incoming fire that has to come from somewhere other than the survey — from geometry, from gun-camera film, from what is known about the attacking aircraft. Wald's memoranda were not a one-line witticism; they were a method for estimating survival probability by region, given assumptions of exactly this kind. Whether he ever said anything about engines in particular looks like a later dramatisation of the work.
Put the general form of it plainly. The data you hold was not drawn at random from the population you care about. It was passed through a filter, and the filter's setting depends on the very quantity you are trying to measure. Vulnerability decides who is in the sample; then you estimate vulnerability from the sample. The estimate is not merely noisy — it is biased in a direction you can predict, because the filter systematically removes the cases that would have carried the strongest evidence.
Once phrased that way it stops being an aviation anecdote. The mutual funds you can inspect today are the ones that were not closed for poor performance, so the industry's historical return looks better than the industry was. The founders available to interview are the ones whose companies lasted, so the habits they share may be the habits of anyone who tried, with the failures silently removed. In each case the missing entries are not missing at random — their absence is the strongest signal in the dataset, and the one a spreadsheet cannot display.
The analogy
THE ANALOGY #Imagine judging a river's rocks by what you pull out of a net strung across it. Everything in the net is smooth and rounded, so you conclude the river makes rocks round. But the net has a mesh: angular rocks caught and snapped the strands, and went on downstream. The catch is a perfect record of what the net could hold, which is not the same as a record of what the river carries.
A net's mesh is visible and can be measured directly, whereas the filter on real data is usually invisible and has to be reconstructed by argument — and unlike a net, the aircraft filter does not merely exclude cases, it excludes them at a rate that varies by exactly the region under study.
Clarifying the model
THE MODEL #The correction is not "distrust the data". The returning-aircraft survey is accurate; nobody miscounted a hole. The problem is that it was being read as an answer to a question it does not address. The cure is to model the selection rather than to lament it: write down what determines membership of the sample, and the same records become informative again — observed holes become evidence about survivability, and their absence becomes the strongest evidence of all.
Two honest qualifications. The inference needs the assumption about where fire lands, and that assumption is doing as much work as the statistics; without it, an empty region is ambiguous between invulnerable, unexposed, and lethal. And survivorship bias is one member of a family, not the whole family — selecting on any variable that is downstream of what you are studying will distort the picture, whether the selection is death, dropout, publication, or simply which records somebody bothered to keep. The lesson generalises to a habit rather than a rule: before reading a dataset, ask what had to happen to a case for it to appear in the file.
A picture of it
THE PICTURE #How to readStart at the rounded terminal at the top and follow a single hit downwards. The first diamond is the filter, and it is not a step anyone performs — it is the world sorting cases by the very property being estimated. The left branch reaches the survey and enters the record; the right branch ends at the double circle, which is a dead end in the data, and the dashed back-edge into the map marks the entries whose absence the map cannot show. The second diamond is the only genuine choice in the picture: read the same map as where fire lands and you get the red outcome, read it as where fire was survived and you get the green one.
What became clearer
WHAT CLEARED #The holes were never a map of danger; they were a map of what an aircraft could take and still fly home. Nothing in the record was wrong, and no more data of the same kind would have helped, because the defect was in how the record came to exist rather than in its contents. Survivorship bias is the general case: whenever entry to a dataset depends on the outcome under study, the missing rows carry the sharpest information, and the visible rows will mislead you confidently and consistently until the filter itself is written into the analysis.
Where to go next
ONWARD #- How censoring is handled formally in survival analysis, where the cases that leave the study early are modelled rather than dropped.
- Why publication bias produces the same distortion in scientific literature, and what funnel plots and registered reports do about it.
Key terms
TERMS #| Term | What it means |
|---|---|
| Survivorship bias | distortion arising when the cases available for study are those that passed a filter related to the outcome being studied. |
| Selection on the dependent variable | choosing cases by their outcome, which severs the link between the sample and the population. |
| Censoring | observation of a case that ends before its outcome is known, which statistical methods can accommodate if it is modelled explicitly. |
| Missing not at random | absence from a dataset that is itself informative, rather than an accident of collection. |
Every term the collection defines is gathered in the glossary.