Releasing to a few first

How do you find out whether a change is safe for everyone without exposing everyone to it?

There is a small logical knot at the heart of shipping software. You want to know whether a change is safe for all your users. The only environment that can answer that question honestly is the one with all your users in it. But putting the change in front of all your users is precisely the act you were trying to justify.

The usual escape is to test beforehand, and testing is worth doing — but notice what it cannot reach. Tests exercise the behaviour you thought to write down, on data you constructed, at volumes you chose. The failures that reach production are, almost by definition, the ones nobody thought of. So the question is not "how do we avoid production" but something stranger: how do we take a small, deliberate bite of the real thing and learn something trustworthy from it?

Begin with the crudest version. Send the change to one server out of a hundred and watch. If it falls over, one percent of requests were affected, and you turn it off.

That plainly limits damage. But does it tell you the change is safe? Only if two conditions hold, and they are worth separating because teams routinely satisfy one and assume the other.

The first is exposure: the sample has to actually encounter whatever would break. If a bug only fires for users with a currency you rarely see, or on a code path used by administrators, one percent of ordinary traffic may never touch it. A canary that never runs the risky path returns a clean result that means nothing at all. So the sample must be representative in the dimensions that matter — and knowing which dimensions matter is a judgement, not a formula.

The second is detection: you must be able to see the difference. Here is where the reasoning gets more interesting than it first appears. Suppose your baseline error rate is one in a thousand and the change doubles it. On one percent of traffic, over ten minutes, how many extra errors do you expect to observe? Possibly a handful. Possibly none. The signal is small because the sample is small, and small samples are noisy — the very thing that makes the canary safe is the thing that makes it hard to read.

That tension is the real content of the idea. Sample size, exposure time, and the size of effect you can detect are locked together: halve the traffic and you need roughly four times as long to detect the same effect at the same confidence, because noise falls with the square root of the count. A team that ships to one percent for five minutes and declares success has usually not measured anything; it has merely failed to notice a catastrophe, which is a much weaker claim.

So how do you make the reading sharper? Compare, rather than threshold. Absolute numbers drift with the time of day, a marketing email, a partner's outage. Run the new and old versions side by side and compare only against each other, and most of that shared noise cancels — which is why the strong form of the practice serves both concurrently rather than checking today against yesterday. It is the same reasoning that puts a control group in a trial.

And then the part that is not statistics at all. A canary is only useful if something acts on it. If the rollout advances on a schedule while a human is meant to be watching a dashboard, you have built a delay, not a safeguard.

One honest limit: some failures are invisible at one percent by their nature. Anything that only appears under full load, anything that corrupts data slowly, anything that depends on a cache warming across the fleet — a canary will pass all of them cleanly. It bounds the damage from the failures it can see and offers no protection at all from the others.

A few refinements.

The first is a distinction often blurred. A canary is a subset of production traffic on the real system. A staging environment is a copy of the system with synthetic traffic. The canary's whole value is that its inputs are genuine — real users, real data shapes, real concurrency — which is exactly what staging cannot supply. They are not two grades of the same thing.

The second concerns who gets picked. Routing by user rather than by request matters more than it sounds: if a single user's session bounces between old and new versions, they may see an item added under one behaviour and priced under another. A consistent hash on user identity keeps each person on one side of the experiment, which is both kinder and cleaner to measure.

The third is the metric you watch. Error rate and latency are the obvious choices and they are necessary but shallow. The failures that hurt most are usually semantic — orders placed but not recorded, a payment path that silently drops a currency — and they show up as a drop in a business quantity, not a spike in a technical one. A canary analysis that only watches HTTP status codes is watching the wrong instrument for the most expensive class of bug.

• Feature flags, which are usually the mechanism that decides who is in the canary.
• Blue/green and rolling deployments, and what each does that a canary does not.
• Sequential testing, which is how you decide a canary early without inflating false positives.

Every term the collection defines is gathered in the glossary.