Connection exhaustion before CPU
A Socratic walk-through of connection exhaustion before CPU — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a service designed to scale infinitely take down the database before it works up a sweat?
An autoscaling service meets a traffic spike and does what it was designed to do: it adds instances. CPU across the fleet sits below half, memory is fine, and then the database starts refusing connections, every instance fails, and the outage is total.
The strange part is the direction. The component that scaled is healthy; the component that did not scale is dead; and the scaling is what killed it. That inversion — more capacity producing less service — is worth taking apart, because it is not a bug in any one piece of code and it will not appear on the dashboard people are watching.
Reasoning it through
REASONING #Begin with a question about units. What does an instance consume from a database? Not queries, which are transient. It consumes connections, and a connection is long-lived and stateful, existing whether or not a query is in flight.
Now ask how each side sizes that resource. The application sizes it per instance: a pool of, say, ten, chosen so one instance can serve its own concurrency comfortably — perfectly reasonable in isolation. The database sizes it globally: one ceiling on simultaneous connections, set from the server's memory and process budget. Postgres allocates a backend process per connection with real per-connection memory, so that ceiling is a genuine resource limit, not an arbitrary setting.
Put the two together and the arithmetic is unforgiving. Total demand is instances multiplied by pool size, and only one of those numbers is controlled by an autoscaler that has never heard of the database. Ten instances at ten connections is a hundred, which is fine. Four hundred is four thousand, against a ceiling commonly in the low hundreds. The autoscaler did not misbehave; it responded to a signal carrying no information about the constraint it was about to break.
Now the second question, which explains the low CPU. Why did the fleet grow to four hundred instances if it was not busy? Because latency rose, and a scaling policy read rising latency as insufficient capacity. But ask why latency rose: requests were waiting for a connection, and waiting is not working. Waiting consumes no CPU. So the fleet was slow and idle at once, and each instance the autoscaler added opened its own pool, taking more of the resource that was already the bottleneck.
That is a reinforcing loop, and it is the actual mechanism: contention raises latency, latency triggers scaling, scaling raises contention. It inverts the assumption behind autoscaling — that adding capacity relieves pressure. When the bottleneck is downstream and shared, adding capacity is the pressure.
A third thing is worth naming, from queueing theory. Little's law says the average number in a system equals arrival rate times average time in system, so the connections you actually need is throughput times query duration — not concurrency, and not instance count. A workload doing a thousand queries a second at five milliseconds each needs about five connections in aggregate. The four thousand were never working; they were idle sockets held open by pools sized in ignorance of each other.
Which points at the fix. If the requirement is a property of the aggregate workload, the allocation must be made at the aggregate — which is what an external pooler does. Put PgBouncer or RDS Proxy between fleet and database, let each instance connect to it freely, and let it multiplex those onto a few real backend connections. Serverless platforms have their own versions, because instance count there is the least predictable number in the system.
The second half is admission control. If demand exceeds what the database can serve, something must wait, and the only question is where. A bounded queue in front of the pool — short timeout, explicit rejection when it fills — fails a fraction of requests and keeps the database alive. Waiting by opening more connections fails everything. Shedding load must be deliberate, because almost every stack's default is to queue without limit.
The analogy
THE ANALOGY #Think of a building where every team gets its own set of ten keys to the same server room, and the room holds four people. With three teams nobody notices. As the company grows nobody revisits the rule, because ten keys per team was never wrong for a team — and one day forty teams hold four hundred keys to a room with four chairs. The queue outside is long enough that management concludes the company needs more staff, and each new team is issued its own ten keys.
keys are inert and a crowded corridor does not damage the room, whereas each open connection consumes real memory on the server — so the excess does not merely wait, it degrades the resource everyone is waiting for.
Clarifying the model
THE MODEL #Three refinements.
First, the pattern is not specific to databases. Any shared downstream with a hard concurrency ceiling behaves this way: a rate-limited API, a broker with a connection cap, a file handle limit, an ephemeral port range on a NAT gateway. The database is simply the most common case, because its ceiling is low and its failure immediate.
Second, "the pool is too big" is usually the wrong diagnosis. Ten per instance is fine at ten instances and fatal at four hundred, so the wrong number is the product, and no per-instance setting is correct across a range of fleet sizes. Hence the durable fixes are architectural rather than a smaller number in a config file.
Third, on monitoring: this outage surprises people because the standard scaling signals cannot see it. CPU, memory, and request rate look healthy while requests queue for a connection. The metrics that would have shown it are pool wait time, pool saturation, and the database's connection count against its ceiling — wait time being the leading indicator, since it rises before anything is refused.
A picture of it
THE PICTURE #How to readThe bars are instances multiplied by a per-instance pool of ten — what the fleet opens if nothing stops it; the flat line is the database's ceiling, which does not move when the fleet does. The two cross around forty instances, and everything right of that crossing is demand the database cannot honour however idle the CPU is. The shape is the point, not the figures: one quantity is under an autoscaler's control and the other is fixed, so the crossing is a certainty to plan for rather than a risk to avoid.
What became clearer
WHAT CLEARED #The failure inverts the usual intuition because the scaling signal and the actual constraint are measured in different currencies. An autoscaler watches CPU and latency; the database rations connections; and requests waiting for a connection are slow without being busy, so the signal says "add capacity" exactly when capacity is the poison. Each new instance brings its own pool, so total demand is a product in which only one factor is under anyone's control, and the loop closes: contention raises latency, latency raises instance count, instance count raises contention. The way out is to stop letting each instance allocate independently — pool at the aggregate, where the requirement actually lives — and to decide deliberately where excess demand waits, since the alternative to shedding a few requests is losing all of them.
Where to go next
ONWARD #- How Little's law estimates the connections a workload genuinely needs, from throughput and query duration alone.
- What transaction-level pooling gives up — session state, prepared statements, advisory locks — and whether an application can tolerate it.
Key terms
TERMS #| Term | What it means |
|---|---|
| Connection pool | pre-opened database connections held by one instance and reused across requests. |
| max_connections | the database's global ceiling, bounded by per-connection memory and process cost. |
| External connection pooler | a proxy such as PgBouncer or RDS Proxy multiplexing many client connections onto fewer real backend ones. |
| Little's law | the average number in a system equals arrival rate times time in system; here, connections needed equals throughput times query duration. |
| Load shedding | rejecting excess requests quickly so the shared resource stays available for the rest. |
Every term the collection defines is gathered in the glossary.