Configuration drift
A Socratic walk-through of configuration drift — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a machine nobody has touched slowly stop matching its own blueprint?
Two servers are built from the same blueprint on the same morning, given the same job, and left alone. A year later they behave differently. One handles a traffic spike; the other falls over. Nobody deployed anything unusual to either of them, and no one remembers changing anything.
The comfortable explanation is that someone did change something and forgot. That is often true, but it is not sufficient — and it is worth noticing why. Even if you could prove no human logged in, the machines would still diverge. So the interesting question is not who touched it but: what is changing a system that nobody is changing?
Reasoning it through
REASONING #Start by listing what can alter a running machine's state without an operator's intent. A package manager applies a security update, and the new version ships a slightly different default. A certificate expires and is renewed with a different chain. A log directory fills and something is deleted to make room. A process restarts and rereads a config file that a colleague edited three months ago but never applied. A kernel parameter is bumped during an incident at 3 a.m. and never recorded. A DNS record the machine caches changes upstream.
None of those is a mistake exactly. Each is a small, locally reasonable event. But now ask the question that makes drift a concept rather than a list: what do those events have in common?
They are all unrecorded state changes to a long-lived thing. And unrecorded changes have a property that recorded ones do not — they cannot be undone, because nobody knows they happened. So the effect does not average out over time. It accumulates. A machine's state is a running total of every event it has survived, and there is no mechanism pulling that total back toward the blueprint.
That is really the whole mechanism, and it is the same shape as any ratchet: changes are easy to make and effectively impossible to reverse, so the difference only ever grows. Notice the corollary — the longer a machine lives, the further it drifts, and the rate does not need to be high for the total to become large. This is why "pets versus cattle" is not just a joke about naming: lifetime is the variable that matters.
Now push on the two-servers puzzle. If both received the same events, they would drift together and still match each other. They diverge because the events are not identical: the update landed on one at a moment the other was busy, one had the incident and the other did not, one's disk filled first. Drift is not just distance from the blueprint; it is distance from the sibling, which is what makes a fleet of supposedly identical machines quietly become a fleet of unique ones.
And here is the consequence people actually feel. Once the machines differ in ways nobody has written down, testing loses its power. A change verified on one host tells you less about the others, because you no longer know what the others are. The failures that follow look random — works here, fails there, cannot reproduce — which is exactly the signature of an environment whose state is unknown rather than wrong.
What stops it? Two answers, and they are genuinely different. The first is continuous reconciliation: a loop that periodically compares the machine to the declared state and corrects the difference. That works, but only for the properties the declaration actually covers — and a config-management tool typically models a few dozen resources out of the thousands of facts a real machine has, so it constrains a slice and leaves the rest free to drift.
The second answer is to attack the lifetime instead of the changes. If a machine is replaced from a fresh image every few days, drift has no time to accumulate — not because anything is corrected, but because nothing survives long enough to add up. That is what immutable infrastructure means, and it is a strictly stronger guarantee, since it does not depend on having modelled the right properties.
The analogy
THE ANALOGY #Think of a hiker's compass bearing. Each step is aimed as accurately as a person can manage, and each is off by a fraction of a degree — sometimes a rock, sometimes a slope, sometimes an unconscious lean. No single step is a mistake. But there is nothing in walking that corrects an earlier error, so the small deviations compound, and after enough kilometres the hiker is a valley away from where the bearing said. Two hikers who start together and take slightly different steps end up apart from each other as well as from the line.
a hiker's errors are random and might partly cancel, whereas most configuration changes are deliberate one-way improvements — a patch, a raised limit, a fix — so the accumulation is a ratchet rather than a random walk, and it never wanders back toward the blueprint by luck.
Clarifying the model
THE MODEL #Three refinements.
First, drift is not corruption and usually not error. Most drifted state is better than the blueprint at the moment it was applied — the patch was needed, the limit was too low. The problem is not the quality of the change but that it exists only on the machine and not in the description, so it is invisible, unreproducible, and lost the moment the host is rebuilt. Drift is an information problem before it is a correctness problem.
Second, detection and prevention are separate capabilities and confusing them is common. Running a reconciliation tool in report-only mode tells you the size of the gap; running it in enforcing mode closes it. Many organisations have the first and believe they have the second, which is how you get a drift report nobody reads and a fleet that keeps diverging.
Third, some drift is legitimate and must be excluded deliberately. A machine's SSH host keys, its generated identity, its accumulated data, its current process state — these are supposed to differ from the image and from each other. A model that flags them produces noise, and noise is how a drift report becomes ignorable. Deciding what is allowed to vary is part of describing the system, not an admission of defeat.
A picture of it
THE PICTURE #How to readRead the four branches as a chain of argument rather than a list of topics. The first two are the inputs — ordinary events, some human and some not. The third branch is the mechanism that turns those inputs into a growing gap, and it is the only branch that explains why drift is inevitable rather than merely common. The last branch answers the third on its own terms: one leaf adds the restoring force that is missing, one removes the lifetime over which the total builds up, and one keeps the model honest about what was never meant to match.
What became clearer
WHAT CLEARED #A machine drifts because it is a long-lived accumulator of unrecorded changes, and nothing in its ordinary operation subtracts. The changes themselves are mostly sensible; what makes them corrosive is that they live only in the machine, so they cannot be reviewed, reproduced, or reversed. That reframes the fix: you either add a force that continuously pulls state back toward a written description, or you stop letting any machine live long enough for the total to matter.
Where to go next
ONWARD #- Immutable infrastructure, and what it costs to rebuild rather than repair.
- The snowflake server, and why the drifted host that survives everything is the one you fear most.
- Drift detection in declarative cloud tooling, and why a plan that shows unexpected changes is a good day.
Key terms
TERMS #| Term | What it means |
|---|---|
| Drift | the accumulated difference between a system's actual state and the state its configuration describes. |
| Immutable infrastructure | an approach in which running systems are never modified in place but replaced from a rebuilt image. |
| Snowflake server | a long-lived host whose accumulated hand changes make it unique and effectively impossible to reproduce. |
Every term the collection defines is gathered in the glossary.