Why the average recovery time lies

Why is the mean time to recover almost never the time it takes to recover?

A team reports a mean time to recover of fifty-two minutes. It goes into the quarterly review, it is compared against last quarter's forty-eight, and someone concludes that recovery is getting worse.

Before accepting that, ask a smaller question. Of the incidents behind that number, how many actually took about fifty-two minutes?

Very often the answer is none. Most were resolved in under ten minutes, one ran for eleven hours, and fifty-two is a number that describes no incident that ever happened. So what exactly is it describing — and is it entitled to be compared quarter to quarter at all?

Start with what a mean is for. It is a balance point: the value at which deviations above and below cancel. When values cluster symmetrically around a centre — adult heights, say — that balance point sits where the data sits, and the mean is a fair summary of a typical case.

So: do incident durations cluster symmetrically? Think about what bounds them at each end. Below, there is a hard floor — an incident cannot take negative time, and realistically not less than the minutes needed to detect and act. Above, there is no bound at all. An incident can run for hours when the cause is unfamiliar, the expert is asleep, or the failure is in a vendor you can only email.

That asymmetry is structural, not incidental. A duration bounded below and unbounded above cannot be symmetric, so the distribution is right-skewed: a dense pile of short incidents and a long thin tail. Empirical studies of repair times generally find something of this shape, often described as approximately log-normal or heavy-tailed — though the precise family varies by organisation, and I would claim no more than the skew itself, which is robust.

Now put the mean back onto that shape. Where does the balance point of a right-skewed pile sit? Pulled right, toward the tail. It lands in the sparse region between the ordinary incidents and the exceptional one — describing neither. That is the first sense in which the average lies: it is not a typical value, and on this shape it structurally cannot be.

Then the second problem, which is worse. How stable is that number?

Because the tail is unbounded, one long incident can move the mean by more than every short incident combined. Fifteen incidents at eight minutes and one at eleven hours produce a mean near fifty; remove that one incident and the mean falls to eight. So a quarter-to-quarter comparison of MTTR is largely a comparison of whether a bad one happened, close to a coin flip on small counts. Most teams have few enough incidents that the sampling noise exceeds any real change in capability.

A third problem hides beneath both. What is being averaged together? A certificate expiry, a bad config push, a corrupted replica, and an upstream provider's outage are not draws from one process — they are different populations with different recovery mechanics. Averaging across them produces a number whose movement can be entirely explained by a change in the mix of incident types. A team that got faster at everything can report a worse MTTR simply by having had more of the hard kind.

And a fourth, quieter one. The measurement itself is discretionary: recovery time is the gap between two human-recorded timestamps, and both ends are soft. When did it start — when the alert fired, when the first user was affected, or when the deploy went out? When did it end — when the error rate recovered, or when the last customer was made whole? Those choices can change a number by a factor of two without anyone acting dishonestly.

So what should be reported instead? The distribution, not a point. The median tells you what a typical recovery looks like and is barely moved by the one eleven-hour incident. A high percentile tells you about the bad case, which is usually what leadership is actually worried about. And plotting the durations themselves communicates more than any summary, because it shows the shape a single number was hiding. Where a summary is unavoidable, the honest form is a median with a percentile beside it and the incident count stated, so the reader can see how little the number is standing on.

Three refinements.

The first is that the mean is not defective; it is being asked the wrong question. A mean is exactly right for a total: mean duration times incident count gives total downtime, a genuinely useful quantity for reliability targets. What it cannot do is stand for a typical case on a skewed distribution. Use it for aggregation, not description.

The second corrects a tempting fix. Removing the eleven-hour incident as an outlier makes the number stable and meaningless — on a heavy-tailed distribution the extremes are not contamination, they are the phenomenon. Those are the incidents that produced the customer harm. Trimming them yields a comfortable metric that no longer tracks what anyone cares about.

The third is about what the metric is for. MTTR is generally used as a proxy for organisational capability, yet recovery duration is dominated by factors that vary independently of capability: how novel the failure was, what time of night it started, whether the fix needed a third party. The underlying components move more honestly — time to detect, acknowledge, and mitigate — because each is shorter, more frequent, and less contaminated by the rare hard case. Improving detection shows up there long before it shows up in the average.

• Log-normal and heavy-tailed distributions, and how to recognise one from a sample.
• Time to detect, acknowledge, and mitigate as separate, better-behaved measures.
• Why the same argument applies to mean latency, and what percentiles fix and do not fix.

Every term the collection defines is gathered in the glossary.