Silent data corruption
A Socratic walk-through of silent data corruption — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does keeping three copies of a file still not guarantee you can get the original back?
Three copies feels like a complete answer: two could be lost and one would remain. Yet the archivist's nightmare is not the disk that dies — that one announces itself. It is the file that comes back a decade later, opens without complaint, and is subtly wrong. So the assumption worth questioning is not how many copies but what a copy is evidence of.
Reasoning it through
REASONING #Start with what a vote actually requires. Three copies protect you by majority: read all three, and if one disagrees, the other two outvote it. Notice how much that sentence smuggles in. It requires that you read all three — and almost nobody does; the second and third copies are typically written once and never opened until the day the first one fails. It requires that a disagreement be visible, and here is the trouble. A dead drive is self-announcing: no flow, no response, an error code. A flipped bit is not. It returns cheerfully, in the right place, with no signal attached to it. There is nothing to notice.
So the first requirement is a detector, not more copies. And a detector means information that did not come along with the data: a checksum, computed from the contents, stored somewhere the corruption is unlikely to reach. Without one, three disagreeing copies leave you with three candidates and no verdict.
Now push on independence, because that is what makes the majority argument work at all. How were the three copies made? Almost always by copying — from the original, or worse, from each other. Draw the timeline and the fault model falls out. Anything that damaged the bytes before the fork is in all three, faithfully replicated, and the vote is unanimous and wrong. Bad memory in the machine that did the encoding, a firmware bug on the controller, a truncated write that the application never checked: none of these are storage failures at all, and no number of downstream copies addresses them. Redundancy multiplies whatever it is given.
Then ask why the per-hop checks that already exist do not close this. They are real and they are good: Ethernet frames carry a 32-bit CRC that catches every burst error up to 32 bits long, and disk sectors carry their own error-correcting code. But each guards exactly one leg of the journey and is stripped and recomputed at every handoff. A bit flipped inside a switch's memory, after the incoming frame was verified and before the outgoing frame's checksum was calculated, is re-blessed: it leaves with a perfectly valid new CRC over corrupted contents. The end-to-end argument in system design says the check that matters is the one computed where the data was born and verified where it is finally used, because only that one spans the parts nobody is watching. The weaker end-to-end checks in the stack are thin — TCP's is a 16-bit sum, so roughly one corrupted segment in 65,536 passes it by chance.
The analogy
THE ANALOGY #Think of three notarised copies of a contract, sealed in three vaults in three cities. Against fire it looks unassailable. But all three were typed by the same clerk from the same draft, and nobody has read any of them since. If the clerk mistyped a figure, the vaults hold three identical false documents; if one page later rots, you cannot tell which vault is honest, because all you can do is compare three copies against each other. What rescues you is something small and separate — a line in the original register giving the contract's total — against which any single copy can be checked alone.
A reader of a contract understands its contents and might notice a figure that makes no sense, whereas a corrupted block carries no meaning that could look wrong; and paper degrades visibly, while a flipped bit leaves no trace of ever having been anything else.
Clarifying the model
THE MODEL #The correction is not that redundancy is useless — it is that it answers a different question than people think. Copies solve erasure: a fault where you know which piece is missing. That is the common case, and erasure coding and RAID are built precisely on knowing which shard is absent. Corruption is the other fault model: the piece is present, plausible, and wrong, and you do not know which one. Redundancy without detection converts an erasure system into a guessing system.
Which is why the practical answer has three parts rather than one. Store a checksum computed at the source, separately from the data it describes, so any single copy can be judged on its own. Scrub: read every copy on a schedule and verify it, so latent damage is found while the other copies are still good, rather than at restore time. And restore-test, because a backup nobody has ever read is a hypothesis. Filesystems like ZFS and Btrfs fold the first two into the storage layer itself, checksumming every block and repairing from a good replica on read.
This is the same independence problem that reliability engineering meets with common-cause failure — two pumps sharing one power bus are not two pumps — but with an extra turn of the screw. A pump's failure is loud, so redundancy only has to survive correlation. A bit's failure is silent, so redundancy has to survive correlation and supply the detector, and the detector must sit outside the path that failed. Adding channels raises the first floor; only independent verification lowers the second.
One honesty note: how often this happens is genuinely disputed, and varies by orders of magnitude between consumer hardware, enterprise drives, and long network paths. The case for defending against it does not rest on a rate. It rests on the failure being undetectable by default, which means an unmeasured rate is not a low one.
A picture of it
THE PICTURE #How to readThis repurposes the packet-layout family, which normally shows network header fields, to lay out the eight-byte footer the T10 protection standard attaches to each data sector — read it left to right, then wrap, with the numbers being bit positions. The first field is the detector: a checksum over the sector's contents, so a copy can be judged alone rather than by vote. The last field records where this sector was supposed to live, which catches the failure mode a content checksum cannot — a perfectly intact block written or returned at the wrong address.
What became clearer
WHAT CLEARED #Copies answer "what if a piece goes missing", and answer it well. They do not answer "what if a piece comes back wrong", because a vote needs both a reason to hold it and voters that could not all have been corrupted by the same event — and copying, by its nature, propagates whatever was already wrong. What recovers your file is not the third copy. It is a small independent statement about what the file should be, checked end to end, and looked at before you need it.
Where to go next
ONWARD #- How erasure codes such as Reed-Solomon get the durability of three copies at a fraction of the storage, and what fault model they assume.
Key terms
TERMS #| Term | What it means |
|---|---|
| Silent data corruption | data returned successfully but differing from what was written, with no error reported anywhere. |
| End-to-end argument | the principle that a correctness check belongs at the endpoints that care, because intermediate checks cover only their own hop. |
| Scrubbing | periodically reading and verifying stored data, so latent damage is found while redundancy still exists to repair it. |
| Erasure | a fault in which a piece is known to be missing, as distinct from corruption, where it is present but wrong. |
Every term the collection defines is gathered in the glossary.