Delivered versus retained

Why does keeping a message after someone has read it change what the system can do?

A classic queue deletes a message once a consumer acknowledges it. A log-structured stream keeps it for a configured period — seven days, thirty, forever — regardless of who has read it.

Described that way it sounds like a storage policy, the sort of thing an operations team tunes and nobody else thinks about. But teams that adopt a retained log end up building differently: they add consumers casually, they reprocess history to fix bugs, they rebuild databases from scratch. Teams on a delete-on-ack queue do none of those things, and not because they lack imagination. A storage setting should not change how people design. Working out why it does is the question.

Ask what a queue's broker is for. It hands out work and tracks what remains to be done. Deleting on acknowledgement is the correct behaviour for that job: a completed task should not sit in the inbox, and the depth of the queue is a meaningful measure of backlog. The broker owns the progress information, because progress is the thing it manages.

Now change one rule — keep the message after it is read — and follow what has to change with it. If the message stays, the broker can no longer use deletion to mean "done". So it needs some other record of how far each reader has got. That record is an offset, a position in the log, and the natural place for it is with the consumer, since only the consumer knows what it has finished. The single change to retention has forced progress tracking out of the broker and into the reader.

What does that make possible? First, a second reader. It has its own offset, so it can read the same messages at its own pace without affecting the first — which is why adding a consumer to a stream requires no change to the producer or to any existing consumer. In a delete-on-ack queue the first reader consumed the message; the second one never sees it, and giving it a copy means the producer or the broker must be told in advance to fan out.

Second, and more consequentially: a reader can move its offset backwards. The messages are still there. That single ability is what all the interesting behaviour rests on. Ship a bug in a consumer, fix it, rewind, and the corrected code sees the same history. Add a service six months late and it can start from the beginning rather than from now. Rebuild a corrupted read model by replaying the log that produced it. None of these is available if the messages were deleted on delivery, because the information no longer exists anywhere — the queue is not a record of what happened, it is a record of what is outstanding.

So the retained log stops being a transport and becomes a source of truth, and that is the real shift. Notice the direction of the dependency it creates: a derived database is now reconstructible, which means it can be treated as a cache rather than as an original. That is a genuinely different posture toward one's own data, and it is why the term event sourcing keeps company with this design.

But there is an honest ordering constraint, because this is where people over-claim. Replayability is only as strong as the ordering guarantee, and ordering is normally per partition, not across the whole topic. Two events on different partitions have no defined relative order, so a replay reproduces the past faithfully only within a partition — which is why the choice of partition key quietly determines what can be reliably recomputed later.

And retention costs. Keeping seven days of everything is real storage, and it means messages you would rather not keep persist for their full window — an awkward property when a deletion request arrives for personal data. Log compaction, which keeps only the latest record per key, is a partial answer: it makes the log a durable snapshot of current state rather than a full history, which trades away the ability to replay the past for bounded storage.

Which suggests the honest summary. Retention is not a better setting; it is a different job. If the broker's role is dispatching work, deletion is correct and its bookkeeping is simpler. If its role is publishing facts that several parties will interpret in ways not all decided yet, retention is what keeps those future decisions available.

Three clarifications.

Retention is not the same as durability. A queue can be entirely durable — fsynced, replicated, surviving any crash — and still delete on acknowledgement. Durability is about not losing a message before it is handled; retention is about keeping it after. Systems are routinely one without the other.

Fan-out is not the distinguishing feature either, though it is often given as one. A topic in a classic pub/sub broker delivers to every current subscriber, which is fan-out with no retention: a subscriber that was offline missed it entirely, and a subscriber that did not exist yet cannot ever see it. Retention is what makes subscription independent of time, not merely of count.

And a consumer that owns its offset owns a new failure mode. Nothing external now stops it committing progress past a message it did not actually handle, which is a way to lose messages that a delete-on-ack queue structurally prevents. The responsibility moved along with the capability.

• Partition keys as the real determinant of what a replay can faithfully reproduce.
• Log compaction, and what it gives up in exchange for bounded storage.
• Handling deletion requests for personal data in a system whose log is the source of truth.

Every term the collection defines is gathered in the glossary.