Tail latency
A Socratic walk-through of tail latency — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a service that answers almost every request instantly still feel slow to use?
A team ships a dashboard. The metrics are excellent: average response time nine milliseconds, ninety-nine percent of requests served in under ten. By any reading of that graph the service is fast. And the users say it is slow — not always, but often enough that they say it without qualification.
Someone will suggest the users are exaggerating. But hold onto the numbers as given, because they are not in conflict with the complaint. The question worth asking is what a user experiences, and whether that is the same event the graph counts. A graph counts requests. A person waits for a page. If those are not the same unit, a service can be genuinely fast per request and genuinely slow per page, with nobody lying.
Reasoning it through
REASONING #Consider how one page is assembled. It is rarely one call. A search result asks a hundred shards for their best matches; a profile page gathers a name, a photo, a permission check, a recommendation set, a notification count. Suppose a page needs answers from a hundred components, each independently fast: one in a hundred requests to a component is slow.
When is the page fast? Only when every one of the hundred is fast. Not most. All. So the probability of a fast page is 0.99 raised to the hundredth power, about 0.366. Roughly sixty-three percent of pages wait on at least one slow component.
Sit with that. Every component is meeting a ninety-ninth-percentile target any engineer would sign off, and nearly two in three page loads are slow. This is the arithmetic at the heart of Jeff Dean and Luiz Barroso's "The Tail at Scale", and the general form is easy: with N independent components each slow with probability p, the chance of at least one being slow is 1 minus (1 minus p) to the power N. At p of one percent, ten components give about ten percent; a hundred give about sixty-three.
Notice what this does to the average. If a page waits on the maximum of a hundred draws, the user's experience is a function of the components' tails, not their means. Improving every component's average by twenty percent may barely move the page; halving one component's ninety-ninth percentile may transform it. Averages mislead here not because they are computed wrongly, but because the user is not sampling the average — they are sampling the worst of many draws.
Where does the tail come from? Rarely from bad code on the slow path. It comes from what a component shares with the rest of the machine and the fleet: queueing behind another request, contention for a shared cache or disk, a background maintenance daemon, a garbage-collection pause, a CPU dropping out of a high-power state, a periodic re-replication. These are ordinary, correct, necessary activities, and they are hard to remove. So if you cannot eliminate the tail, what can you do?
Stop waiting for it. Dean and Barroso's central move is that a system can be made tail-tolerant the way it is made fault-tolerant: not by making slowness impossible, but by arranging that one slow component cannot hold the answer hostage. A hedged request sends the same query to a second replica after a short delay — typically the time by which the great majority of responses would already have arrived — and takes whichever answer comes first, cancelling the other. Because the hedge fires only for the small minority already late, the extra load is a few percent while the effect on the worst percentiles is dramatic. A tied request goes further: enqueued on two replicas at once, each copy carrying the identity of the other, so whichever server begins work first tells the other to drop it. That removes the waiting period entirely, at the cost of a brief window in which both might start.
The analogy
THE ANALOGY #Think of a dinner that cannot be served until every dish is ready. Each cook is reliable — ninety-nine times in a hundred the dish lands on time. With one cook the dinner is nearly always punctual. With a hundred dishes it is served when the slowest cook finishes, and the odds that none of the hundred hit a problem are poor. The kitchen's average performance is superb and the dinner is regularly late.
A kitchen's delays are largely independent, whereas real servers share power, switches, storage and maintenance schedules, so their slow moments are correlated — which makes the tidy independent-probability arithmetic an approximation rather than a measurement, sometimes optimistic and sometimes pessimistic.
Clarifying the model
THE MODEL #Three clarifications, because "tail latency" gets used loosely.
First, fan-out and chaining are different shapes with different arithmetic. When a request fans out in parallel, the latency is the maximum of the set and the tail dominates. When services are chained in series, the latencies add, so the means add — but each hop contributes its own chance of a bad draw, so the composed tail is still worse than any single hop's. Both punish tails; only the parallel shape does so through the maximum.
Second, percentiles do not compose the way people expect. No operation turns component ninety-ninth percentiles into a system ninety-ninth percentile, which is why the only trustworthy number is measured end to end, at the boundary the user experiences. A dashboard of per-service percentiles can look uniformly green over a system that is not.
Third, the honest limit of the arithmetic above: it assumes independence, and that assumption is doing real work in the result. Correlated stalls — a shared top-of-rack switch, a fleet-wide deployment, a hot shard everyone queries — break it. And hedging carries risk of its own: if slowness is caused by overload, duplicate requests add load to a system already struggling, which is why hedges are rate-limited and tied requests carry an explicit cancellation. Tail-tolerance is a set of trade-offs, not a fix.
A picture of it
THE PICTURE #How to readRead top to bottom as time. One user request enters at the root and becomes several subqueries, and the root can reply only when all have answered — so the page's latency is the slowest arrow, not the average. Shard one returns promptly while shard two stalls on a pause it did not choose; the hedge to its replica is the intervention, fired only after the normal wait has passed, and the page completes on whichever copy answers first. The final arrow is the original slow answer arriving too late to matter, which is exactly what tail-tolerance is aiming for.
What became clearer
WHAT CLEARED #A service is fast per request and slow per page whenever a page needs many components at once, because the user waits on the maximum rather than the mean. At one percent slowness and a hundred components, roughly sixty-three percent of pages are slow while every component's dashboard looks excellent — so averages, and even per-component percentiles, understate what people experience. Since the causes of the tail are ordinary shared-machine activity that cannot be engineered away, the workable answer is not to prevent slowness but to stop depending on it: hedged and tied requests let a system finish without waiting for its unluckiest component.
Where to go next
ONWARD #- Why queueing theory predicts latency exploding as utilisation approaches saturation, and what that implies for headroom.
- How micro-partitioning and selective replication let a system move load off a component that has turned slow.
Key terms
TERMS #| Term | What it means |
|---|---|
| Tail latency | the high percentiles of a latency distribution, such as the 99th or 99.9th, as opposed to its mean or median. |
| Fan-out | one incoming request spawning many parallel sub-requests whose answers must all be collected. |
| Hedged request | a duplicate sent to a second replica after a short delay, with the first answer taken and the other cancelled. |
| Tied request | a request enqueued on two replicas at once, each carrying the other's identity so whichever starts first cancels its twin. |
Every term the collection defines is gathered in the glossary.