Order within a slice

Why can a system promise to keep messages in order only within one part of itself?

Read the guarantee a log-structured broker actually offers and it has a strange shape. Messages arrive in the order they were sent — within one partition. Across partitions, nothing is promised at all.

That looks like an implementation detail leaking into a contract. Ordering is such a simple idea; why would anyone deliberately guarantee it in fragments? So start with the harder question underneath: what would it cost to promise it everywhere?

Ask what "before" even means for two events on two different machines. There is no shared clock accurate enough to settle it, so the only way to say one came first is to run both past a single arbiter that assigns them numbers. A single arbiter is a single append point — one log, one writer. Whatever that one writer can sustain is now the entire system's throughput, permanently, no matter how many machines you buy.

So a global order is not expensive because implementers were lazy. It is expensive because order is a scarce, serialized resource: to have it you must make everything queue at one place. Parallelism is the exact opposite move — several independent sequencers, each numbering its own arrivals, with no defined relationship between their numbers.

And notice that storage is only half of it. Suppose the broker did stamp a global sequence number on every message. Two consumers reading concurrently would still process message 7 and message 5 at whatever pace their machines managed. Order is a property of the whole path — write, store, read, process — and it survives only if every stage of that path is single-threaded. A guarantee that ends at the disk is not a guarantee.

Given that, what is the most order you can buy without giving up parallelism? Here is the useful observation: you rarely need every event ordered against every other. You need events about the same thing ordered. Two updates to account 4471 must not be swapped. An update to account 4471 and one to account 9902 have no relationship worth defending.

That is the whole design. Slice the stream so that everything about one entity lands in one slice, by hashing a chosen key to a partition. Now each partition is its own tiny single-arbiter system, and you get as many of those as you have partitions. Order per key; throughput per partition count.

Two things then quietly break the promise, and both are worth naming. First, the mapping is hash(key) modulo partition count, so adding a partition changes where existing keys land — and no broker goes back and re-shuffles history. An entity's story splits across two partitions at the seam, with no order across it. That is why partition count on a keyed topic is closer to permanent than adjustable.

Second, order can be lost inside a single partition by the producer itself. If a client has several batches in flight and one is retried after a failure, a later batch may land first. Kafka's fix is a per-partition sequence number on each batch, with the broker rejecting anything out of sequence; that idempotent producer became the default in Kafka 3.0 and holds ordering with up to five requests in flight. Before that default, a great many topics had a reordering hazard nobody had noticed.

Third, and most commonly, a consumer receives records in order and immediately hands them to a thread pool. The broker kept its promise; the application threw it away.

The misconception worth correcting is that partition order is about delivery. It is about the sequence in which a single reader is handed records, and it stops mattering the moment that reader parallelizes. If you need concurrency within a partition, the honest pattern is to fan out by key again inside the consumer, so each sub-worker still owns a whole key — not to spray records across workers and hope.

The second refinement: per-key order is weaker than it sounds. It is not causal order. If your invariant spans two entities — move money from one account to another, reserve stock before confirming an order — partitioning gives you nothing, because the two events are in different slices by construction. The available answers are to choose a key that covers the whole invariant (both events keyed by the transfer, not by either account), or to use a transaction, or to design the consumer to tolerate arrival in either order. There is no partitioning scheme that makes this problem go away.

And global order is still purchasable: a topic with one partition gives it. Nobody is hiding a better option. You are simply paying the price the first paragraph named, which is why the guarantee is written in fragments rather than made optional.

• How a transactional producer extends atomicity across partitions, and what it still refuses to promise about order.
• Why hot keys turn the ordering guarantee into a throughput ceiling for one entity, and what to do when one key is most of the traffic.

Every term the collection defines is gathered in the glossary.