Chaos engineering

Why deliberately break a working system that is already surviving?

Somebody proposes killing a production server on purpose, during business hours, while customers are using the system. The obvious objection writes itself: the thing is working, so leave it alone.

Hold that objection up to the light for a moment, though. It contains a claim — that the system is working — and the question worth asking first is how we came to believe it. Usually the evidence is simply that nothing bad has happened lately. But "no failure has occurred" and "the system tolerates failure" are different propositions, and only one of them has been tested.

Consider what the redundancy in a large system actually consists of. A failover replica. A retry policy. A circuit breaker that sheds load when a dependency is sick. A cache that is supposed to serve stale data when the database is unreachable. Every one of those is code, or configuration, that runs only when something has already gone wrong.

So ask: when was that code last executed? For much of it, the honest answer is never, or once, eighteen months ago, before four refactors. Recovery paths are the least-exercised part of any system, and they are also the part you are staking everything on. Would you expect untested code to work first time, at three in the morning, under load, in an unfamiliar situation? We do not believe that anywhere else in software.

Now push on the word "surviving". A system that has not fallen over might be tolerant of failure, or it might merely not have been hit yet — a replica that has silently been out of sync for a month looks identical, from the outside, to one that is ready. That is the important asymmetry: from the absence of failure you cannot distinguish robustness from luck. The only way to tell them apart is to supply the failure yourself.

And here is the argument that turns this from a curiosity into a practice. The failure is coming regardless. A machine will die, a network will partition, a dependency will go slow. Your only real choice is when — and there is an enormous difference between a failure that happens at eleven on a Tuesday morning, on one instance you selected, with the whole team watching and a rollback ready, and the same failure at three on a Sunday, everywhere at once, discovered from a customer's tweet. Chaos engineering is not the manufacture of risk. It is the relocation of an existing risk into conditions where you can afford it.

There is a further gain, and it is the one practitioners usually rate highest. What breaks during a controlled experiment is frequently not the thing you injected. You kill an instance to test failover, and you discover that failover works fine but the alert never fired, or that the runbook refers to a dashboard that was deleted, or that traffic drained correctly and then a downstream service retried so aggressively it collapsed something else. Those are the interaction failures nobody enumerates in advance, and they surface only when a real component is genuinely absent.

The discipline has a specific shape, and it matters, because "break things and see" is not it. The published principles — from the group at Netflix who formalised the practice after the Chaos Monkey tool of 2011 — start with a steady-state hypothesis: a measurable statement of normal, in terms of business output rather than internals, such as the rate at which video streams start. Then you predict that the steady state will hold under the injected event, inject it into a bounded slice of the system, and compare. If the steady state holds, your confidence is now evidence-backed. If it does not, you have found a weakness cheaply, with your hand on the switch.

The bounded slice is the load-bearing part. Blast radius — how much can go wrong if the experiment goes worse than expected — is chosen, not discovered, and every serious programme keeps an abort mechanism that can stop the experiment in seconds. Teams also do not start here. The usual progression is a tabletop exercise, then a staging environment, then one instance in production, then a fraction of traffic, and only much later something like a whole region. And an honest caveat: the case for all this rests on practitioner reports and post-incident experience rather than controlled studies, since running the counterfactual organisation is not possible.

Three clarifications.

The first corrects the name. "Chaos" suggests randomness for its own sake, and the practice is closer to the opposite: a hypothesis, a bounded injection, a measurement, a conclusion. It is an experimental method, and the discipline is in the boundaries rather than the breakage.

The second is about production. The insistence on experimenting in production strikes people as recklessness, but it follows from the reasoning above: staging has different traffic, different data volumes, different scale, and no real users — so a failure survived in staging tells you about staging. That said, "production" is the destination of a programme, not its entry point, and running there before you have observability, an abort mechanism, and an on-call team who agreed to it is the recklessness people fear.

The third is what a passing experiment is worth. It does not prove the system is resilient; it proves this failure mode, in this configuration, on this day, was survived. Systems drift, so the result decays — which is why the practice is continuous and automated rather than an annual event.

• Steady-state hypotheses: how to define "normal" in terms customers would recognise.
• Why retry storms and timeout mismatches are among the most common findings.
• Game days as the human half of the practice: exercising the responders, not just the servers.

Every term the collection defines is gathered in the glossary.