Lineage

When a number on a report is wrong, how do you find every other number that is now also wrong?

Someone tells you that a column in one table has been wrong since Tuesday. A currency conversion was applied twice, say. You fix the column. Are you done?

Almost certainly not, and the reason is worth sitting with. That column was not sitting still while it was wrong. Things read it. Those things produced other things, which were read in turn. The error has been copying itself forward for days, and the copies do not look like errors — they look like ordinary numbers on ordinary dashboards. So the real question is not "what broke" but "what did the break touch", and answering it requires knowing something that no single table knows about itself.

Start with a smaller version. Suppose you have exactly two tables: raw orders, and a daily revenue summary built from it. If raw orders is wrong, is the summary wrong? Obviously. And you knew that not by inspecting the summary's contents but by knowing where it came from. That knowledge is a fact about the relationship between two objects, not a fact inside either of them.

Now notice what that implies. If you delete every dashboard and every query and keep only the tables, the information you need has vanished, even though all the data is still there. The dependency is not a property of the data. It is a property of the transformations — the SQL, the pipeline definitions, the notebook that someone ran once. Lineage is the name for that dependency graph, made explicit and kept somewhere you can query it.

Why does it have to be explicit? Could you not just work it out when you need it? In principle yes: read every query in the organisation, parse it, see which tables it reads and which it writes. In practice, that is precisely how the good lineage tools work — they parse SQL and pipeline definitions and build the graph mechanically. The reason it has to be built ahead of time and stored is not that it is impossible to derive, but that you need it at exactly the moment you are least able to spend three days deriving it: during an incident.

Here is the step that changes how people think about it. Ask yourself which direction you want to walk the graph. When you are debugging — "why is this number wrong?" — you walk upstream, toward sources, narrowing until you find the transformation that introduced the fault. When you are containing — "who else is affected?" — you walk downstream, and the graph fans out rather than narrowing. Those are the same graph traversed in opposite directions, and they answer completely different questions. The downstream walk is what people mean by impact analysis, and it is the one that keeps growing, because every consumer of a wrong table is itself a table someone else consumes.

Does that fan-out ever stop? At the edges, yes: eventually you reach things nobody reads — a report, an export, a model's training set. But two things make the boundary softer than it looks. The graph can contain feedback, where a derived table is joined back into an upstream process, so the fault can revisit ground you thought you had cleared. And the last hop is often invisible: a spreadsheet somebody downloaded, a figure pasted into a board deck, a decision taken on Wednesday. Lineage tooling can see as far as its parsers reach and no further, and human copies out of the system are where it goes blind.

There is a second limit worth naming honestly, because it is easy to miss. Most practical lineage is recorded at table granularity: this table feeds that table. But the wrong currency column may not be used by every downstream query. Column-level lineage exists and is genuinely more precise, but it is harder to derive — a SELECT *, a wildcard expansion, or a user-defined function can defeat the parser — so many systems either lack it or have it partially. The practical consequence: table-level lineage tends to over-report impact, which is the safer failure but not a free one, because an over-broad blast radius costs you credibility the third time you tell forty teams their numbers might be wrong.

Three refinements pull the picture together.

The first is that lineage is metadata about code, not about data. It changes when someone edits a query, not when a row arrives. That is why it goes stale in exactly one way — an unregistered pipeline, an ad-hoc script, a manual load — and why coverage, not accuracy, is the usual weakness. A lineage graph is rarely wrong about what it shows; it is usually silent about something it never saw.

The second is about time. Real damage assessment needs the graph as it was during the bad window, not as it is now. If a transformation was rewritten on Thursday, the current graph describes Thursday's dependencies and may mislead you about Tuesday's. Systems that version their lineage can answer this; many cannot, and that limitation is worth knowing before you rely on the answer.

The third corrects the most common misconception: that lineage tells you what to fix. It does not. It tells you what to check, in what order, and it gives you the list you would otherwise have to reconstruct from memory under pressure. The judgement — which of these consumers actually cares, which need a correction notice, which can simply be rebuilt — stays human. The value of the graph is that the list is complete rather than remembered.

• Column-level versus table-level lineage, and what defeats a SQL parser.
• Why lineage and data contracts tend to arrive together in a platform's history.
• Backfills: how you re-run a downstream graph in dependency order without doing it twice.
• The blind spot at the human edge — exports, spreadsheets, and decisions already taken.

Every term the collection defines is gathered in the glossary.