Backpressure

Why does a system that never asks a fast producer to slow down end up slower than one that does?

Put a queue between a fast producer and a slow consumer and nothing is refused: every request is accepted, the consumer works flat out. Now build the rival system, the one that makes the producer wait whenever the queue is full. It looks strictly worse — it says no to work it could have taken.

Yet the second finishes more useful work per second than the first. Not merely lower latency: more completed work. Neither system got a faster consumer. So where does the first one's capacity go?

Begin with the only thing that is certainly true. Over any interval, work completed cannot exceed capacity multiplied by time. A queue does not change that; it changes when work happens, never how much can happen. So a buffer absorbs variance — a burst arriving faster than the mean, drained during a lull — and nothing else. If arrivals persistently exceed service, the backlog grows by the difference every second, and no buffer size fixes it. Size only sets how long you have before you find out.

Say the consumer can do 700 requests a second and 1,000 arrive. The queue grows by 300 a second: after a minute, 18,000 wait. Little's Law gives the consequence — the average number in the system equals arrival rate times average time in it, so waiting time is backlog over throughput: 18,000 divided by 700, about 26 seconds. Every request now waits 26 seconds, including those that would have been answered in five milliseconds.

Here is where capacity starts vanishing. Ask what a client does after 26 seconds. It has a timeout, and it abandoned this request long ago. The consumer is nonetheless working the queue in order, spending full effort computing answers no one is waiting for. That work is paid for and thrown away. Then the client retries — which is an arrival, so the arrival rate rises. More arrivals, longer queue, longer wait, more abandonment, more retries. The loop closes on itself, and goodput — completed work someone still wants — falls while the machine stays pinned at a hundred per cent.

A smaller drain runs alongside: the queue itself consumes memory the consumer needs, and in a garbage-collected runtime a deep queue of live objects raises tracing cost, so the collector gets more expensive precisely when you are behind.

Now the falsification test, because "wasted work" is a claim that can be wrong. It predicts something specific: take the saturated system and change only one thing — discard, at dequeue, any item whose deadline has already passed — and useful completions per second should jump immediately, with no extra capacity added. That is exactly what dropping expired work does in practice. And the refuting observation is just as sharp: if latency improved but goodput stayed flat, then nothing was being wasted, the loss was pure capacity, and this account would be wrong.

There is a second test in the other direction. If deadlines are load-bearing, a pipeline without them should degrade gracefully — and it does. A nightly batch job just buffers and catches up. Nobody abandons, nothing is wasted, the backlog is only a delay. Collapse needs an impatient client.

So what is backpressure? It is making the queue's fullness into a signal that travels upstream, to somewhere that can stop producing: a bounded queue that blocks its writer, a receive window, a credit scheme, an explicit request for n items and no more. The saving is not politeness. It is that the system now learns it is over capacity at the moment work is admitted, when refusing costs nothing, instead of discovering it after the work has been paid for.

That exposes the real design question. The signal must reach something that can actually slow down. If the ultimate producer is external — users, a market feed, sensors — it cannot be slowed, and backpressure degenerates into shedding at the edge. The honest decision is then not whether to drop but which to drop, and no amount of buffering will answer that for you.

The misconception worth naming is that a bigger buffer is a cheap partial fix. It is not a fix at all; it is a longer fuse. A buffer sized for a burst is engineering; a buffer sized in the hope of covering a rate mismatch converts a fast, visible failure into a slow, confusing one, and adds latency to every request meanwhile.

Two honest qualifications. First, "backpressure" names a family, not a mechanism: blocking a writer, refusing an item, shrinking a window and shedding load have quite different fairness and deadlock properties, and a blocking scheme propagated back through a shared thread pool can stall unrelated work. Second, unbounded queues remain the default in many runtimes and messaging systems, largely because a bounded queue forces the designer to answer "what happens when it is full?" while an unbounded one always looks fine in a demo. That is installed-base inertia, not merit.

This sits next to congestion collapse, the same conservation law where the contended resource has no owner and no one can be signalled; backpressure is the version available when there is an upstream to tell. It differs from tail latency, where capacity is adequate and rare slow components are amplified by fan-out.

• How credit-based flow control differs from blocking, and why one can deadlock a shared thread pool where the other cannot.
• Why serving the newest request first can raise goodput under overload even though it is grossly unfair.

Every term the collection defines is gathered in the glossary.