Pascal's mugging

Why can a vanishingly unlikely promise of enormous reward hijack a perfectly rational calculation?

A stranger stops you and says: give me ten pounds, and tomorrow I will use powers you have no evidence of to grant you a quadrillion days of happiness. You do not believe him. But how much do you not believe him? Not zero — you cannot honestly assign zero to a claim you merely find absurd.

And there the trouble begins, because the standard advice for deciding under uncertainty is to weigh each outcome by its probability and take the option with the highest total. If the reward is large enough, that rule seems to tell you to hand over the money. Nick Bostrom set this out in a 2009 note in Analysis, building on a puzzle Eliezer Yudkowsky had posed a couple of years earlier. The unsettling part is not that the mugger might be telling the truth. It is that the rule you trust appears to recommend paying him.

Let us try the obvious escape first. Surely we just say the probability is so small that the product stays tiny? Then let us make it concrete and watch the two quantities move.

Suppose the mugger claims a thousand days. You might grant that one in a hundred. He raises it to a million. Does your credence fall by a factor of a thousand to match? Try it honestly: a man claiming a million days does not seem a thousand times less likely to be telling the truth than a man claiming a thousand — he seems roughly as implausible, maybe a little more. Now he claims ten to the hundredth power. He said one extra sentence. Your credence should fall, yes — but it falls by some factor, while the payoff has just been multiplied by ten to the ninety-fourth.

Do you see the shape of the difficulty? The reward grows exponentially in the length of the description, while any sane rule for discounting claims can only make your credence fall by something like a constant factor per extra bit of description. Adding "to the power of a hundred" costs him a handful of characters and multiplies his offer beyond anything the shortening of his sentence could offset. Formalize the discount as strictly as you like — assign each hypothesis a prior falling with its shortest description, as the Solomonoff-style priors do — and the mugger simply names a number that is enormous and short to describe, of which there are plenty. The expected value climbs without limit as he talks.

So the mugger does not need to be believed. He needs only for your credence to shrink more slowly than his stated reward grows, and the arithmetic of exponentials is such that he can always arrange this by naming a bigger number. He can rob you with a sentence.

Now, is this just Pascal's Wager over again? Not quite, and the difference matters. Pascal's infinite reward can be resisted by objecting that infinities do not behave like quantities in arithmetic. The mugger's offer is finite. Every step of the calculation is ordinary. That closes the usual escape hatch.

What is the honest state of the problem? Unresolved. Several repairs have been proposed. You can bound how much value you are willing to assign to any outcome, which caps the payoff but has awkward consequences elsewhere. You can argue that a claim to be uniquely positioned to affect an enormous number of people should carry a penalty proportional to that number, since if such vast populations exist, the chance that you are the pivot for them is correspondingly slim. You can insist that hypotheses supported by no evidence be discounted separately from hypotheses that are merely improbable. Each of these has defenders and each has costs, and Bostrom himself concluded that no fully satisfying resolution was in hand. I would not present any of them to you as the answer.

The most common misreading is that this is a story about gullibility, and that a sensible person simply refuses. Of course they refuse. The puzzle is not what to do — everyone agrees you keep your ten pounds. The puzzle is that expected-value maximization, which is otherwise a very good rule and the backbone of decision theory, apparently disagrees with the verdict everybody holds. When a trusted rule and a firm intuition part company, one of them is wrong, and the interesting work is finding out which.

The second misreading is that the answer is just to round tiny probabilities down to zero. Consider what that costs you. Small probabilities of large harms are how we reason about pandemics, asteroid strikes, and nuclear accidents — cases where the payoff is genuinely enormous and the probability genuinely small, and where taking the product seriously is the right response. Any rule that disarms the mugger by ignoring small probabilities also disarms those. The difficulty is drawing a principled line between a small probability with evidence behind it and one conjured to order by whoever wants your money.

That, I think, is the real lesson lurking here: the vulnerability is not in the multiplication. It is in the fact that one party gets to name both numbers.

• How the leverage penalty works, and whether it generalizes beyond the mugging case.
• Why bounding utility solves this but creates trouble for reasoning about very large futures.
• Where the line falls between a mugging and a legitimate small-probability catastrophe.

Every term the collection defines is gathered in the glossary.