Event time versus arrival time
A Socratic walk-through of event time versus arrival time — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #How does a system that never stops receiving data decide it has seen enough to answer?
You are asked for a simple number: how many orders were placed between nine and ten this morning. With a database and a day's hindsight, this is a one-line query. With a live stream of order events, it is genuinely hard — and the hardness has nothing to do with volume or speed.
Here is why. At 10:00:00 your system has seen some orders stamped between nine and ten. At 10:00:03 it sees another one, stamped 09:59:58, delayed by a phone that lost signal in a lift. At 10:04 another arrives, stamped 09:47, from a device whose upload retried for seventeen minutes. So when, exactly, is the nine-to-ten answer finished? The stream does not end. Nothing tells you the last straggler has landed.
Reasoning it through
REASONING #Notice first that there are two clocks in play, and confusing them is the whole problem. One is when the thing happened — the timestamp the phone wrote when the customer tapped Buy. The other is when we found out — the moment the event reached the processing system. In a well-behaved world these differ by milliseconds; in the real world the gap is unbounded and irregular.
So which clock should the window use? Suppose we use arrival time — count everything that lands between nine and ten. That is trivially computable and answers a question nobody asked. Re-run it tomorrow on the same recorded stream and you get a different number, because the delays will be different. A result that depends on network conditions is not a business measurement. So the window has to be defined in event time, and now we own the problem: we must group by a clock whose values arrive out of order.
Can we simply wait until we are sure? Ask what "sure" would require. It would require knowing that no device anywhere still holds an unsent event stamped before ten. Nothing in the system can know that — a phone in a drawer might deliver in a week. Complete certainty demands unbounded waiting, and unbounded waiting means never answering.
So the question changes shape. It is no longer how do we know we have everything; it is how long do we choose to wait, and what do we do about what arrives after. That is a trade, and it has exactly the axes you would expect: wait longer and the answer is more complete but later; answer sooner and it is timelier but more likely to move.
The mechanism that carries this decision through a streaming system is the watermark. A watermark is an assertion injected into the stream itself: I believe I have now seen everything with an event time earlier than 09:55. It is not a fact and it is not derived from proof. It is usually a heuristic — typically the highest event time observed so far, minus a chosen allowance for lateness. When a watermark passes the end of a window, the window fires and emits its result.
And what of the event that arrives after its window fired? Three honest options exist and systems like Apache Flink and the Beam model expose all of them. You can drop it, and accept a small undercount. You can hold the window's state open for a declared grace period and emit a corrected result — which requires every downstream consumer to be able to accept a restatement. Or you can route it to a side output so a human or a batch job can deal with it. Which one is right is a business decision, not a technical one: a fraud alert would rather be fast and revised, a regulatory report would rather be late and final.
The analogy
THE ANALOGY #Think of the ballot count in a postal election. Each vote carries the date it was posted — its event time — and arrives at the counting house whenever the post delivers it, which is its arrival time. Counting by arrival date would produce a number that tells you about the postal service rather than the electorate, so the rule is defined on the postmark. But nobody can prove the last postmarked ballot has arrived, so the returning officer declares a deadline: anything postmarked by Thursday and received by Monday counts. That deadline is the watermark. Ballots that arrive on Tuesday are the late events, and the law has to say in advance whether they are discarded, counted in a supplementary declaration, or set aside for review.
an election has a genuine final ballot and a moment when the count really is complete, whereas a stream has no last event at all — so a streaming watermark is a standing bet renewed continuously, not a one-off cut-off.
Clarifying the model
THE MODEL #Three refinements are worth making.
The first is that a watermark is not a timer. It is easy to read "wait five minutes" as wall-clock patience, but a watermark advances with the data, not with the clock on the wall. That is what allows the same job to replay a year of history from a log and produce identical results: the stream carries its own sense of time. If it were wall-clock based, a replay would fire every window instantly and empty.
The second: a watermark that stops advancing stalls everything behind it. If a system reads several partitions and one goes quiet, the overall watermark is the minimum across them, so a single idle source can freeze all output. Real systems handle this with explicit idleness detection, and it remains one of the more common operational surprises in streaming.
The third, most often glossed over: widening the lateness allowance reduces dropped events but never eliminates them, and it costs state — every open window must be held for the whole grace period. There is no setting that gives completeness and low latency together; the honest design is to pick a point on that curve deliberately.
A picture of it
THE PICTURE #How to readRead downward as elapsed real time and each message's stamp as its event time — the point is that those two orders disagree. The first exchange is the easy case, where the clocks agree closely. The turn comes at the watermark message: nothing has proved the hour is complete, yet the operator acts on that assertion and fires the window. Then follow the straggler arriving four minutes after its window closed; the branch below it is a choice made in advance, not a fault — either the dashboard accepts a restatement of a figure it has already shown, or the event leaves the pipeline.
What became clearer
WHAT CLEARED #A never-ending stream cannot know it has seen everything, so it never decides that it has. What it decides is when to stop waiting — a watermark is that decision made explicit, advanced by the data rather than by the wall clock, and always accompanied by a stated policy for what happens to whatever turns up afterwards. The question "is the answer complete" quietly becomes "how complete did we choose to make it, and can our consumers accept a correction".
Where to go next
ONWARD #- How the same job can be replayed over historical data and produce identical windows, and what that implies for testing.
- Session windows, where the boundaries are defined by gaps in activity rather than by the clock.
- What it costs to keep window state open, and why grace periods are bounded by storage rather than by principle.
Key terms
TERMS #| Term | What it means |
|---|---|
| Event time | the timestamp recorded when something actually happened, carried in the event itself. |
| Processing time | the moment an event reaches the system that handles it; dependent on network and retry behaviour. |
| Watermark | an assertion flowing through the stream that all events earlier than a given event time are believed to have been seen. |
| Allowed lateness | a declared grace period during which a window's state is retained so late events can produce a corrected result. |
| Side output | a separate stream to which events too late for their window are routed instead of being dropped silently. |
Every term the collection defines is gathered in the glossary.