Running both and comparing

How do you prove a replacement behaves like the original when nobody knows exactly what the original does?

Testing normally starts from a statement of what ought to happen. You write the expected result, you run the code, you compare. That works because someone knows the answer in advance.

Now take the case where nobody does. The system being replaced has run for fifteen years; its documented behaviour is a subset of its real behaviour, and some of what it does was never decided by anyone. You cannot write the expected results, because the expected results are precisely what you are trying to discover. So what evidence is even available?

There is one source of truth still in the room: the old system itself, running, with real requests arriving at it. If it cannot be described, it can still be observed. And that suggests a move — stop trying to state the specification and start sampling it.

So: take each real request, send it to both systems, return the old system's answer to the user, and record the new system's answer for comparison. Nothing the new one produces is used. That single property is what makes this safe enough to run on all traffic at once: a divergence is an observation, not an incident. GitHub's Scientist library packaged exactly this shape in 2016, and the pattern is variously called a parallel run, shadow traffic, or differential testing.

Two requirements decide whether the experiment means anything.

The first is side effects. The new path must not send the email, charge the card, post the ledger entry, or consume the next invoice number. If it does, you have not built a test — you have built a duplicate transaction, and you will discover this from a customer. So the shadow path must run against a sandbox, write to a shadow store, or have its outbound effects stubbed, and that plumbing is usually more work than the comparison itself.

The second is inputs. Both must see the same request and the same state. If the new system reads its own replicated copy of the data, replication lag alone produces divergences that mean nothing about the logic. Time is the sharpest version: two systems reading their own clocks a few hundred milliseconds apart will disagree about anything straddling a boundary.

Now the comparison, which is where the naive version of this idea quietly fails. Equality is the wrong test. Most differences are benign: timestamps, generated identifiers, the ordering of a collection, float formatting, whitespace, a field the new system populates and the old one leaves empty. So you must normalise before comparing — and notice what a normalisation rule really is. It is a claim that some difference does not matter. That growing list of claims is one of the most valuable artefacts the migration produces, and also the easiest place to hide a real bug, which is why the rules deserve review and why raw pairs should be sampled and kept rather than only their verdicts.

Then the harder problem, and it is a genuinely uncomfortable one. The old system is the reference, so its bugs are the standard. A divergence tells you the two differ; it does not tell you which is right. Every difference has to be triaged into three buckets — the new one is wrong, the old one is wrong, or neither matters — and only the first is a defect. The middle case is a decision, not a finding: someone with domain knowledge must decide whether to reproduce the old behaviour faithfully or to record an accepted, dated difference that says "we are deliberately changing this". Undocumented, it will resurface as a bug report in six months.

Finally, the limitation no engineering removes. Production traffic covers what actually happens, weighted by how often it happens. Year-end processing, a refund of a refund, the one customer with the odd configuration — these may not occur during the whole comparison window. A parallel run bounds risk in proportion to frequency rather than importance, which is backwards for the rare-and-catastrophic cases. The complements are replaying historical data, especially period-end batches, and targeted tests for the rare paths you already know about; neither covers the ones you do not.

Three refinements.

First, this is evidence, not proof. What a clean parallel run establishes is that on the distribution of traffic observed, over the window observed, after the normalisations declared, the two systems agreed. Each of those clauses is a real limit, and stating them is what makes the result honest rather than reassuring.

Second, the exit criterion should not be zero differences. It should be a difference rate that is fully explained — every remaining class understood, categorised, and either accepted with a decision or fixed — held stable across a window long enough to include the business cycle, which usually means a month-end and often a year-end. A team chasing zero tends to widen the normalisation rules until the number reaches zero, which is the same as switching the test off.

Third, a misconception worth correcting: a parallel run is not a rehearsal for cutover. It exercises the comparison, not the switch. Whether traffic can actually be moved — and moved back — is a separate question answered by routing a small share of live traffic to the new system and watching real users, which is a different risk with a different failure mode.

• Traffic replay from logs, and reproducing the state a request originally saw.
• Canary releases, which answer what a parallel run cannot: can traffic actually be moved.
• Property-based testing, for the rare paths production traffic never exercises.

Every term the collection defines is gathered in the glossary.