Choosing an indicator

Of everything a service could measure, how do you pick the number that stands for the user's experience?

A service emits four hundred metrics. CPU, memory, garbage collection pauses, connection pool depth, cache hit ratio, thread counts, queue lengths, error counts by a dozen classifications. All are true, and all are cheap to collect.

Now suppose you must pick one number, put it on a page, and let a team's decisions turn on it — whether to ship, whether to stop and fix, whether to wake someone at three in the morning. Which of the four hundred do you choose?

The uncomfortable observation is that almost none of them qualify, and the reason is not that they are inaccurate. It is that the thing you actually care about — whether the service is working for the person using it — was never one of the four hundred in the first place.

Start by asking what a good indicator is for. Not "what is happening inside the machine" — that is diagnosis, which the four hundred metrics already serve. The indicator's job is different: it must stand in for a judgement no instrument can make directly, namely whether a user is being well served.

Which means an indicator is a proxy, always. So the question becomes: what makes one proxy better than another?

Try a candidate. CPU utilisation at eighty percent — does that tell you whether users are happy? It does not, in either direction. A service can be at ninety percent and serving everyone perfectly, or at five percent and serving nobody because it is deadlocked. The correlation with user experience is weak, and weak in a way that varies — so a threshold that is meaningful this quarter misleads next quarter, after an optimisation changes the relationship.

Now try another. Request success rate, measured at the point where the user's request enters your system. Ask the same question: if this drops, is a user worse off? Yes, essentially by definition — a failed request is a failed interaction. That directness is the first property worth naming. A good indicator sits as close to the user as you can put it, so that a movement in the number means something happened to somebody.

But directness alone is not enough. Consider a service where every request succeeds and every one takes forty seconds. Success rate is a perfect hundred percent. Was the user well served? Obviously not — so a single indicator usually cannot cover a journey, and standard practice is a small set: availability and latency, plus quality or freshness where the service's value depends on them. A search service that returns quickly with degraded results fails in a way neither availability nor latency detects.

Here is the move that organises all of this. Rather than starting from the metrics you have, start from the journeys a user actually takes — log in, search, add to basket, pay — and ask of each: what would it mean for this to have gone badly? The answers become your indicators. That reversal is the practical heart of the discipline, because the four hundred metrics were generated by the system's internal structure while the journeys are generated by the user's purpose, and only the second has any authority over what counts as good.

Two more properties to test each candidate against.

Can you aggregate it honestly? An indicator is usually expressed as a ratio of good events to valid events over a window, because that form is comparable across time and volume in a way a raw count is not. And the percentile matters: a mean latency conceals the tail entirely, which is why practice reaches for something like the 95th or 99th percentile — with the caveat that percentiles do not average across services, so a p99 of p99s is a well-known mistake rather than a shortcut.

And is it actionable? If the number degrades, can this team do something about it? An indicator dominated by a third party you cannot influence produces alarm without agency, and trains people to ignore it. This is a genuine tension, not a rule: the user's experience does include that third party, so the honest resolution is usually to measure it but hold the team accountable only for the part it controls.

One caveat worth stating plainly. Where you measure changes what you see. Server-side measurement misses everything between your edge and the user — DNS, the network, the client's rendering — so it systematically flatters you. Client-side measurement captures the real experience but arrives incomplete, because the most broken sessions are the ones that fail to report. That gap is real, not an artefact to be tuned away.

Three refinements.

The first separates two things that are often merged. An indicator is a measurement: the ratio of successful checkout requests to valid checkout requests over five minutes. An objective is a target placed on that measurement: 99.9 percent over thirty days. Choosing the indicator is the hard, judgement-laden part; the target is a business negotiation that only becomes meaningful once the measurement is trustworthy. Picking a target first is how teams end up defending a number that measures the wrong thing.

The second corrects a common instinct: that more indicators means better coverage. A page of thirty cannot be reasoned about during an incident, and it quietly reintroduces the problem you were solving — nobody knows which number decides anything. A handful per critical journey is the working scale, and the discipline is in what you leave off.

The third is about drift. The relationship between an indicator and real user experience is an assumption, not a fact, and deployments erode it. A new client-side cache can hold your server-side success rate steady while users see stale data. So the indicator needs periodic re-validation against complaints and abandonment — evidence from outside the measurement system, the only thing that can tell you your proxy has come loose.

• Turning an indicator into an objective, and who is entitled to set the target.
• Percentiles and why a p99 of p99s is not a p99.

Every term the collection defines is gathered in the glossary.