Blind analysis

Why do physicists deliberately hide their own data from themselves until the analysis is frozen?

A collaboration spends a decade and a great deal of money collecting a dataset, and then goes to considerable trouble to prevent itself from seeing the answer in it. The final number is shifted by a secret offset, or the interesting region of the data is masked, or fake signals are quietly inserted. Only when every analysis choice has been fixed and written down is the veil lifted, in a one-time ceremony that cannot be repeated.

That is a strange thing for careful people to do to themselves. If the worry were dishonesty, hiding the data would not stop anyone determined to cheat. So what exactly is being defended against, and why does self-restriction work better than resolving to be careful?

Begin with a fact from the history of measurement rather than from ethics. Plot the published values of a physical constant against the year they were published and you frequently see the same shape: not a scatter about the truth, but a slow drift, each result sitting close to the last, converging on today's value from one side over decades, with error bars that never quite covered where the value was going. Millikan's electron charge is the famous case Feynman used, and reviews of measured constants hold milder versions.

Ask what produces a drift like that. Not fabrication — the pattern is far too widespread. So the mechanism must be something that operates through ordinary, competent work.

Here it is. An analysis is not one decision but hundreds: where to place a cut, how wide the bins are, which runs to exclude as bad, which background model to fit, how to treat an outlier, when there is enough data to stop. Each of those choices has a defensible answer and often several. Individually, none is wrong. And crucially, most of them can be revisited.

Now add one more ingredient: the analyst can see what each choice does to the answer. Nothing more is needed. If a plausible cut moves the result away from the expected value, it is natural to go looking for a bug — and in a real experiment, looking hard for a bug usually finds something. If the same cut moves the result towards expectation, the search is shorter. Nobody has lied. But the effort spent on scrutiny has become a function of the answer, and that asymmetry alone biases the result towards whatever was expected. Stopping is the sharpest version: an analysis that halts when the number looks right is a measurement of the expectation, not of nature.

So the enemy is not dishonesty; it is the ordinary responsiveness of good judgement to information it should not have yet. Then ask the decisive question: can that be fixed by intending not to do it? The evidence says no. The bias operates through exactly the faculties one would use to resist it, and it leaves no trace in the record, because every individual step was defensible when it was taken.

That is what forces the move to a commitment. If you cannot trust your future judgement in the presence of the answer, remove the answer until the judgement is already spent. The choices get made against something that carries the same statistical structure as the real data but not the real result: an unknown offset added to the final number, a masked signal region, scrambled event associations, injected fake signals, or a small open subset used for tuning while the rest stays sealed. When every cut, model and threshold is frozen and documented, the box is opened — once. Whatever comes out is the result.

Two real examples make the shape concrete. The LIGO collaboration ran blind injections, in which a small group could secretly insert a simulated gravitational-wave signal; the 2010 event nicknamed the Big Dog was analysed to the point of a written discovery paper before the envelope revealed it was planted. The Fermilab Muon g-2 experiment kept the clock frequency needed to convert its measurement into a physical value secret, held in sealed envelopes, and revealed it at a recorded unblinding in 2021 — after which the number could not be adjusted.

Some refinements.

First, blinding does not mean working without feedback. Almost every check still runs while blind: calibrations, resolutions, control regions, closure tests, deliberate cross-checks on samples that contain no signal. What is withheld is narrowly the thing that would let a choice be graded by its effect on the final answer.

Second, unblinding rules do the real work, and they are agreed beforehand: what will be looked at, in what order, and — most importantly — what happens if something looks wrong afterwards. A collaboration that reserves the right to re-tune after unblinding has spent all the effort and kept none of the protection. In practice a post-unblinding change is allowed only for a demonstrable error, and is documented alongside the original number.

Third, this is a family, not a single technique, and it is not free. Blind analyses are slower, and there is a real cost when a frozen analysis turns out to have been suboptimal in a way that only the data could have revealed. The reason the trade is still judged worthwhile is that the alternative error is invisible, and an invisible error cannot be corrected by later work — it propagates into the next experiment's expectation, which is precisely how those historical drifts were built.

• Pre-registration in medicine and psychology, which is the same commitment device under another name.
• How blinding interacts with the look-elsewhere effect and with trials-factor corrections.
• The neutron lifetime discrepancy between beam and bottle methods, a live case where blinding did not resolve the disagreement.
• What a collaboration should do when an unblinded result is clearly wrong.

Every term the collection defines is gathered in the glossary.