Grade earned, not declared
A Socratic walk-through of grade earned, not declared — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does calling a dataset trustworthy do nothing, while making it pass a test does?
A data catalogue offers a small green badge: Certified. Someone with the right permissions ticks a box, and from then on the table is presented to the whole company as trustworthy. Six months later the upstream system changes a column's meaning, the badge is still green, and a hundred people are still relying on it.
Now consider a second arrangement. There is no box to tick. A table carries the badge only while a suite of checks passes on every load — row counts within an expected band, no duplicate keys, no nulls in a column that must never be null, freshness within an agreed window. When a check fails, the badge disappears by itself.
Both are "certification". Why is the second worth something and the first almost nothing? The tempting answer is that the second is more rigorous, but that is not quite it — and finding the real difference is the whole exercise.
Reasoning it through
REASONING #Start with what a badge is for. A consumer of the table cannot inspect it themselves; that is the entire point of the label. So the badge is a substitute for an inspection they will not perform. Which raises the question: a substitute for whose inspection, made when?
In the tick-box design, the badge stands for one person's judgement at one moment. It says: on the day I looked, this looked fine. Is that a claim about the table, or a claim about a past state of the table? Only the second. And data is not a book; it changes every night. A statement about a thing that has since been overwritten a hundred and eighty times is not false so much as stale beyond meaning.
So consider what would have to be true for the badge to keep meaning something. It would have to be re-established as often as the thing it describes changes. And who can afford to re-inspect nightly? Nobody, by hand. Hence the shift: the badge stops being a statement and becomes an output. It is emitted by a check that runs whenever the data does.
Notice what has quietly changed. In the first design, trust flowed from authority — someone senior said so. In the second, trust flows from evidence, and specifically from evidence that is regenerated at the same rate as the risk. Is the second design therefore proving the data is correct? No, and this is the part worth being careful about. A passing suite proves only that the failures you thought to test for did not occur. A column can be within range, non-null, unique, fresh — and wrong, because a source system silently switched from reporting gross to net.
So what is the honest claim? Not "this data is right", but "these specific ways of being wrong have been checked, just now". That is a much smaller claim, and it is worth far more than the large one, because it is checkable and it decays visibly.
There is a second thing a good certification carries, and it is easy to miss. A dataset is not trustworthy in the abstract; it is fit, or unfit, for a particular use. Clickstream data with a two percent loss rate is entirely fit for judging which page design wins and entirely unfit for billing. A badge that does not say for what invites exactly the misuse it was meant to prevent — which is why the useful ones name the guarantee, not the virtue: this table's key is unique, its freshness lag is under an hour, its revenue column reconciles to the ledger within a stated tolerance.
The analogy
THE ANALOGY #Think of the difference between a restaurant with a framed award on the wall from 2019 and one with a hygiene inspection score posted in the window that the council re-issues after each unannounced visit. Both are signals hung on the same door. The award tells you someone was impressed once. The score tells you what a defined set of checks found recently, and it can drop — and the fact that it can drop is what makes the current value informative at all.
an inspector walks in and looks at whatever they find, whereas an automated data check only ever looks for the failures someone wrote down in advance — it will never notice the entirely new way the kitchen has gone wrong.
Clarifying the model
THE MODEL #The first correction: the argument here is not against human judgement. Deciding which checks matter, what the acceptable freshness window is, and which tolerance the revenue column must reconcile within is judgement work of a high order, and no tool supplies it. What automation replaces is not the judgement but the re-assertion of it. A person decides the criteria once; the pipeline re-establishes whether they hold, every run.
The second: a badge that can go red is only useful if going red does something. If a failed check merely writes a line in a log nobody reads, you have rebuilt the tick-box with more machinery. The versions that work either block the bad load from being published at all, or visibly mark the table as stale in the place people find it, or notify the owner with an obligation attached. The signal has to reach the decision it is supposed to inform.
The third, and the one most often skipped: tests decay too. A check that has passed every day for a year may be passing because the condition holds, or because the check has quietly stopped examining anything — a filter that now matches zero rows will happily report zero violations forever. Data-quality tooling has a genuine blind spot here, and the mitigations, such as asserting that a test's evaluated row count is itself non-zero, are more folklore than settled practice.
A picture of it
THE PICTURE #How to readRead the horizontal axis as where the claim comes from and the vertical as how often it is renewed. The bottom-left corner holds signals that are both hand-asserted and never refreshed — the decorative badges. Moving right alone is not enough: a one-time profiling report is machine-produced and still ages into uselessness. Moving up alone is not enough either. Only the top-right corner, where a machine re-establishes the claim at the same rhythm the data changes, gives a signal whose green state carries information today.
What became clearer
WHAT CLEARED #A declaration of trustworthiness is a claim about the past made in the present tense. A passing test is a claim about the present, narrow enough to be checkable and repeated often enough to stay true. The badge earns its meaning not from who granted it but from the fact that it would have gone away on its own — and from naming which specific failures it rules out, and for which use.
Where to go next
ONWARD #- How service level objectives for freshness and completeness are negotiated between a producing team and its consumers.
- Why anomaly detection on data profiles catches different failures than assertion-style tests, and where each is blind.
- What lineage adds: knowing not just that a table failed, but everything downstream that should now be treated as suspect.
Key terms
TERMS #| Term | What it means |
|---|---|
| Data certification | marking a dataset as approved for general use; meaningful only when the mark is tied to conditions that are re-checked. |
| Fitness for use | the principle that quality is relative to a purpose rather than an absolute property of a dataset. |
| Assertion test | a check that a named condition holds, such as uniqueness of a key or absence of nulls in a column. |
| Freshness SLA | an agreed maximum lag between an event happening and it being visible in a table. |
| Reconciliation | comparing a derived figure against an independent authoritative source, such as a general ledger, within a stated tolerance. |
Every term the collection defines is gathered in the glossary.