Killed with memory to spare

Why does a program die for lack of memory on a machine that has plenty left?

A container exits with code 137. The dashboard for the host it was running on shows twenty gigabytes free, flat, all through the incident. Nothing else on the box was under pressure. And yet the kernel killed the process for running out of memory.

The instinct is to distrust the dashboard, or to suspect a leak, or to blame a spike too brief to graph. Those are all real things, but none of them is needed here. The more interesting possibility is that both readings are true at once: the machine genuinely had memory to spare, and the process genuinely ran out. If that is not a contradiction, then "out of memory" must mean something narrower than we assumed.

Start with the question the kernel is actually asking when a process requests another page. Is it "does this machine have a free page?" — or is it "is this requester allowed another page?"

If it were only the first question, a single greedy process could starve every neighbour on a shared host, and containers would be worthless as an isolation mechanism. So there must be a second, narrower accounting. On Linux that accounting is the control group, or cgroup: a hierarchy that charges every page a process touches to a group, and enforces a ceiling on the group's total. A container is, in large part, just a cgroup with a filesystem view attached.

Now the puzzle dissolves into a question of scope. The limit is per-group; the free memory is per-host. A container with a two-gigabyte limit hits its ceiling at two gigabytes regardless of what the host has.

But notice what happens next, because it is where the surprise really lives. When a process on an unconstrained system approaches the physical limit, the kernel reclaims — it evicts clean page cache, writes back dirty pages, swaps. Only when reclaim fails does the killer arrive. The same sequence happens inside a cgroup at its limit, with one crucial narrowing: reclaim runs within that group's own pages. The kernel will not evict a neighbouring container's cache to make room for yours, because that would defeat the isolation the limit exists to provide. So the group can fail to reclaim while the host is comfortable, and the cgroup out-of-memory killer picks a victim inside the group. To the process, this is indistinguishable from a machine-wide famine. Exit code 137 is 128 plus 9 — the shell's convention for a process terminated by SIGKILL, which is what the killer sends.

Two further details explain most of the confusing cases in practice.

The first is that page cache counts toward the limit. Read a large file inside a container and the cached pages are charged to it, even though cache is reclaimable and looks like nothing. Usage climbs toward the ceiling for reasons that have nothing to do with the heap, and a genuinely modest allocation is the one that tips it over — so the killed process is often not the culprit, merely the last claimant.

The second is that the process usually cannot see its own limit. /proc/meminfo is not namespaced; a program reading it inside a container sees the host's totals. So any runtime that sizes itself as a fraction of "available memory" — a JVM choosing a default heap, a database sizing a buffer pool, a worker pool guessing a concurrency level — will size itself against a number that has no bearing on what it is allowed. It then grows confidently into a wall it never knew existed. This was common enough that the JVM added explicit container awareness (available from OpenJDK 8u191 and on by default from JDK 10 through UseContainerSupport), which reads the cgroup limit instead. Many other runtimes still do not.

Which suggests the reframing. "Out of memory" was never a statement about the machine. It is a statement about a budget, and the budget is a deliberate artefact of policy, set by whoever wrote the deployment manifest. The kernel enforced exactly what it was told to enforce.

A few corrections worth making explicit.

The limit is not a reservation. Setting a two-gigabyte ceiling does not set aside two gigabytes; it only caps you. On a host that is genuinely oversubscribed you can still be killed by the host's out-of-memory killer while under your own limit. Two independent mechanisms can end the same process, and telling them apart matters when you are diagnosing.

Not every ceiling kills. Cgroup v2 distinguishes memory.max, the hard limit that triggers the killer, from memory.high, a throttling threshold: crossing it puts the group under heavy reclaim pressure and stalls its allocations rather than terminating anything. A workload that is slow and thrashing rather than dead may be sitting against a high watermark, and no kill event will appear anywhere in the logs to explain it.

And the victim is not always the offender. The cgroup killer scores processes within the group — broadly by memory footprint, adjustable via oom_score_adj — so in a container running several processes it may kill the largest rather than the one whose allocation crossed the line. Reading the kill as an accusation is a good way to spend an afternoon investigating the wrong process.

• How memory.high throttling produces slowness with no kill event to explain it.
• Whether swap accounting should be enabled inside cgroups, and what changes when it is.
• Setting requests and limits so the scheduler's packing decisions match the ceilings it enforces.

Every term the collection defines is gathered in the glossary.