Acceptance sampling

How can testing twenty parts ever justify shipping ten thousand of them?

A lorry arrives with ten thousand bolts. An inspector pulls twenty, measures them, finds none defective, and signs the lot into the factory. The other nine thousand nine hundred and eighty are never touched.

The instinct is that this is theatre. Twenty out of ten thousand is a fifth of one percent — surely a bad lot could hide behind a sample that small without even trying? And yet this procedure has been the backbone of industrial receiving inspection for the better part of a century. So either an enormous industry has been fooling itself, or the sample is answering a different question than the one we assumed. It is worth working out which.

Start by being precise about what the inspector has and has not learned. He has learned nothing whatsoever about bolt number 7,412. That bolt was not measured, and no amount of statistics will measure it retroactively. If the question is is every bolt good, the sample is useless and the instinct is right.

But that was never the question. Ask instead: what kind of lot is this? Now the twenty bolts are evidence, and we can ask how strong.

Suppose the lot is genuinely one percent defective. What is the chance that twenty random bolts are all good? Each has a 0.99 chance of being fine, so it is 0.99 raised to the twentieth — about 82 percent. A clean sample is the expected outcome, and the lot sails through. Fine.

Now suppose the lot is ten percent defective. The same calculation gives 0.9 to the twentieth, which is about 12 percent. So a ten-percent-bad lot passes a clean-twenty test roughly one time in eight, and gets caught the other seven. And a twenty-percent-bad lot passes about one time in a hundred.

Notice what has happened. The sample cannot tell you a lot is perfect, and it never claimed to. What it does is discriminate: it lets a good lot through most of the time and stops a badly bad lot most of the time. It is a filter with known leakage, not a certificate.

Here is the part that reliably surprises people. Nowhere in that calculation did the number ten thousand appear. Lot size barely enters — what governs the sharpness of the test is the size of the sample. Twenty bolts drawn from a hundred thousand carry almost exactly the same evidential weight as twenty drawn from a thousand, because you are estimating a proportion, and a proportion does not know how big its population is. The intuition that "a fifth of one percent is not enough" is measuring the wrong ratio entirely.

So what does buy you a sharper test? More sample. Take fifty instead of twenty, still allowing zero defects, and the one-percent lot now passes only 61 percent of the time — you have made life harder for the good supplier — while the ten-percent lot passes about 0.5 percent of the time. The curve got steeper: better separation between good and bad, paid for with inspection effort and with more good lots wrongly rejected.

Which exposes the honest structure underneath. There is no risk-free plan. Every sampling scheme apportions two errors: the chance of rejecting an acceptable lot, which the producer pays for, and the chance of accepting an unacceptable one, which the consumer pays for. The published standards — the military tables of the 1940s and their civilian descendants — are not magic; they are pre-computed trade-offs, indexed by how much risk each side agreed to carry.

And one caveat worth keeping. Deming argued for decades that when the incoming defect rate is stable and known, sampling is usually the wrong answer: depending on the cost of inspecting a part against the cost of a defective one reaching assembly, you should generally inspect all of them or none of them, and the middle ground is waste. Whether that holds depends on knowing the process is stable — which is itself an inference, made under uncertainty, from a sample.

Three refinements tie the reasoning together.

First, the misconception this most often corrects: a passed lot is not a clean lot. Acceptance sampling makes a probabilistic statement about lots, not a guarantee about units. If any single failure is catastrophic — an aircraft fastener, a sterile implant — sampling is the wrong tool, and you need either full inspection or a process controlled tightly enough that the population argument becomes credible on its own.

Second, the whole scheme rests on the sample being representative. Randomness is not a formality here; it is the entire load-bearing assumption. Twenty bolts scooped from the accessible end of the pallet are twenty bolts about one corner of the pallet.

Third, and most usefully: the deeper move in modern quality practice is that incoming inspection is a poor substitute for knowing the process. A supplier who can show a stable, capable process gives you far better evidence than any spoonful, because then the population you are inferring about is one you understand rather than one you are guessing at.

• Operating characteristic curves, and how to read a sampling plan straight off one.
• Statistical process control as the alternative to inspecting quality in after the fact.
• Sequential sampling, where the sample size is not fixed in advance but stops as soon as the evidence is decisive.

Every term the collection defines is gathered in the glossary.