Replication lag
A Socratic walk-through of replication lag — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why can a system tell you something different one second after it told you it was saved?
You change your display name. The page confirms it. You reload, and the old name is back. A few seconds later it is right again, and it stays right forever.
Nothing was lost — the write succeeded and was never in danger, and nothing was corrupted. Yet for a moment the system contradicted itself, and it did so after telling you the change was saved. That combination deserves an explanation, and the explanation turns out to be the price of something you asked for without noticing.
Reasoning it through
REASONING #Start from why there is more than one copy. A single database is one machine: bounded capacity, and if it fails the data is unavailable. So systems keep replicas — copies kept up to date, serving reads and ready to take over. That is what you implicitly asked for in wanting a system that survives failure and serves more readers than one machine can.
Now the mechanical question. A write lands on the primary; how does a replica learn of it? The primary must send it — as a log record, over a network, to a machine that receives, applies, and durably records it. Every step takes time, so there is an interval in which the primary knows something the replica does not.
Here is the decision that determines everything: when does the primary say "saved"? It can wait until replicas confirm, or answer as soon as it has committed locally.
If it waits, the write takes as long as the slowest replica's round trip, and an unreachable replica makes it block or fail. Such a system has made every write depend on every replica's availability — the opposite of what replicas were for.
If it does not wait, the write is fast and survives a replica being down, and the reload one second later may reach a replica that has not yet applied the change. That is the whole phenomenon: asynchronous replication, and the interval is replication lag.
So the contradiction is not a bug but the visible surface of a choice to make writes fast and independent of replica health. And note what "saved" honestly meant: durably committed on the primary, never "visible everywhere". The interface did not distinguish the two.
What governs the size of the lag? Usually a few milliseconds — but it is a queue, not a constant, and queues behave badly. If the primary produces changes faster than a replica applies them, lag grows without bound rather than settling. Hence spikes cluster around bulk imports, schema migrations, a long query blocking apply, or a cross-region link during an incident. Minutes are possible, and the failure mode is drift rather than a clean break.
What can be done about the symptom — seeing your own change vanish? Notice it is narrower than general consistency: you do not need every reader to see your write immediately, you need you to. That guarantee is read-your-writes consistency, and there are honest ways to provide it: route a user's reads to the primary for a short window after writing; or record the write's log position and have the replica wait until it catches up; or, cheapest, keep the submitted value in the client's session.
A companion anomaly confuses people more. If a client reads one replica and then another further behind, time can run backwards — a comment that was there is gone. That violates monotonic reads; the fix is to pin a client to a replica so whatever it sees, it keeps seeing.
An honest caveat: "eventually consistent" is a genuine guarantee but a weak one. It promises convergence if writes stop and says nothing about how long. Some systems bound the lag, some only report it, and treating the two as equivalent is a common and expensive mistake.
The analogy
THE ANALOGY #Think of a newsroom with a wire desk and several regional printing plants. The story is filed and confirmed at the desk — real, recorded, not going to be lost. But the presses in each region receive it and print at slightly different moments. Ask the plant nearest you before its edition rolls and you are told there is no such story, quite correctly, because there is not one there yet. Wait a moment and it appears.
a newspaper is a discrete edition, so a plant is printing either yesterday's paper or today's, whereas a replica applies changes continuously and can be partly current — it may know your last three writes and not the fourth, a state no printing plant is ever in.
Clarifying the model
THE MODEL #Three refinements, and a warning about the interface.
First, this is not data loss, and the distinction matters when triaging. Lag delays visibility; the write is committed and will appear. Loss occurs only when a primary fails before its acknowledged writes reached any replica — precisely the exposure asynchronous replication accepts for speed, and the reason semi-synchronous modes exist, waiting for one replica but not all.
Second, the choice is not binary: between full synchrony and full asynchrony sit useful positions, such as waiting for a quorum, or synchronously within a region and asynchronously across them.
Third, a common misdiagnosis. "The database is inconsistent" is almost never right: each replica is internally consistent and correct about a different point in the primary's history. That is not corruption but a skew in time, and the reframing points at the right instruments — seconds behind primary, and which replica served the read.
The warning: "your change has been saved" does real damage. It reports durability and is read as visibility. A system with meaningful lag is better off routing the reader who just wrote back to the primary than explaining the difference: the user's model — what I just did should be what I see — is not wrong; it is a guarantee the system can choose to provide.
A picture of it
THE PICTURE #How to readFollow the numbered messages down the page; vertical distance is elapsed time. The confirmation at step 3 is honest — committed and durable on the primary — but the note between the lifelines marks the log record still travelling. The read at steps 7 and 8 crosses that window and reaches the replica before it has applied the change, so the replica answers correctly for its own state and the user sees the old value. Nothing is lost here: the replica's self-directed arrow is it applying the record, after which every later read is stable. The anomaly lives entirely in the gap between the two arrows leaving the primary.
What became clearer
WHAT CLEARED #There is no moment at which a distributed system is simply "up to date" — there is a primary that has committed a change and replicas at various distances behind it. Asynchronous replication buys fast writes that do not depend on any replica's health, and pays with a window in which different copies answer differently. "Saved" describes durability on the primary and never claimed visibility elsewhere. What users need is far narrower than global consistency — to see their own writes, and to not see time run backwards — and both can be provided cheaply.
Where to go next
ONWARD #- What a quorum read actually guarantees, and why R plus W greater than N is the usual rule.
- How change data capture turns the same replication stream into a feed for search indexes and caches, inheriting the same lag.
Key terms
TERMS #| Term | What it means |
|---|---|
| Replication lag | the interval between a change committing on the primary and a given replica having applied it. |
| Asynchronous replication | acknowledging a write once the primary has committed, without waiting for any replica. |
| Semi-synchronous replication | waiting for at least one replica to acknowledge before confirming the write, bounding loss without depending on all. |
| Read-your-writes consistency | the guarantee that a client always sees the effect of its own earlier writes. |
| Monotonic reads | the guarantee that a client never sees a value older than one already seen. |
| Eventual consistency | the guarantee that replicas converge if writes stop, with no promise about when. |
Every term the collection defines is gathered in the glossary.