Bulkheads

Why does deliberately wasting capacity keep one failing feature from taking down all the others?

Suppose you have two hundred worker threads and ten features to serve. The efficient arrangement is obvious: one shared pool, any thread free to take any request. Nothing sits idle while work is waiting. Any other arrangement wastes capacity by definition, because you will sometimes have a queued request for one feature and an idle worker reserved for another.

And yet the recommended arrangement is the wasteful one. Split the pool, give each feature its own reserved slice, and accept that you will occasionally refuse work you could have done. Why would anyone pay that, and what exactly are they buying?

Follow one bad day through the shared pool, slowly.

One of the ten features calls a recommendations service. That service starts responding in thirty seconds instead of thirty milliseconds — not failing, just slow, which is the important detail. Requests for that feature keep arriving at their usual rate, and each one now occupies a worker a thousand times longer than before.

Ask what fraction of the pool that feature holds after a minute. Not its usual tenth. It grows without limit until it hits the pool size, because arrivals continue while departures have nearly stopped. Ten seconds in it might hold half the pool; a minute in, all of it. And now the checkout feature, whose dependencies are perfectly healthy, cannot get a worker. The whole application is down.

Sit with the shape of that. The failure did not spread through a bug, or through shared data, or through anything the healthy features touched. It spread through the resource they had in common. Sharing a pool is what coupled them. That is why this is a structural problem and not a coding one — no amount of correctness in the checkout code protects it.

So what would have prevented it? Only one thing: a limit on how much of the shared resource that one feature could hold. Give recommendations a cap of twenty workers and the slow dependency exhausts twenty workers, then its requests are refused immediately. Checkout never notices. The feature that is broken is the only feature that is broken — which is the entire promise of the pattern, borrowed by Michael Nygard in Release It! from the transverse bulkheads that divide a ship's hull so a breach floods one compartment rather than the vessel.

Now the cost, honestly. What does the cap cost on a good day? Occasionally recommendations has twenty-five requests in flight and refuses five while forty workers sit idle in other partitions. That is real waste, and it is not recoverable — it is the premium being paid. So the design question is not "how do I avoid wasting capacity" but "how much am I willing to waste to bound the blast radius", and the answer depends on how much you would lose if everything failed together.

Notice also that the resource in question is rarely just threads. The same reasoning applies to connection-pool slots, memory, file handles, queue depth, and the request-rate budget of a shared downstream. Any single resource that all features draw from is a coupling, and each needs its own partition or the isolation is incomplete. This is why bulkheading a thread pool while leaving a shared database connection pool untouched often achieves nothing: the compartment has a hole in it.

And there is a stronger version: partition the whole system rather than a resource, running several complete copies each serving a subset of customers. That is cell-based architecture. The refinement on it is shuffle sharding, described in the AWS Builders' Library — assign each customer a random combination of workers rather than a fixed group, so that with eight workers taken two at a time there are twenty-eight distinct pairs and two customers rarely share both. The principle is that overlapping random subsets isolate better than disjoint fixed ones.

The remaining design question is where to put the partitions, and the honest answer is that it is a judgement, not a formula. Partition on the boundary along which failures actually correlate: usually per downstream dependency, sometimes per tenant, sometimes per criticality tier so that a checkout path and a reporting path can never compete. Partition too finely and each slice is too small to absorb a normal burst, so you shed load on ordinary Tuesdays; too coarsely and you have rebuilt the shared pool with extra ceremony.

Three refinements.

The first is that bulkheads do not make anything more reliable. The failure still happens, and the affected feature is still down. What changes is the scope — one feature instead of ten. Availability of the whole was traded for containment of the part, and if you measure only aggregate success rate you may not see the benefit at all; you see it in the incidents that did not become total outages.

The second is that a bulkhead needs a defined behaviour when full, and this is where implementations go wrong. Rejecting immediately is usually right, because the caller can then fall back or degrade. Queueing without a bound is usually wrong: the queue becomes exactly the unbounded shared resource you partitioned to avoid, with the extra insult that requests time out after waiting instead of before.

The third is how bulkheads relate to their neighbours in the resilience toolkit. A circuit breaker stops calling a dependency once it is known to be failing; a bulkhead limits the damage during the interval before the breaker trips, and during every failure the breaker never notices — a dependency that is slow but not erroring, for instance. Rate limiting caps total incoming work; bulkheads govern how that work is divided internally. They overlap, but none substitutes for another, and the slow-dependency case above is precisely the one that only the bulkhead catches.

• Cell-based architecture and shuffle sharding, which apply the same reasoning to whole systems rather than to pools.
• How to size a partition from measured concurrency, using Little's Law rather than intuition.

Every term the collection defines is gathered in the glossary.