The outbox

How do you save a record and send a message about it without either happening alone?

A service takes an order. It writes a row to its database, then publishes an OrderPlaced message so that billing and shipping find out. Two lines of code, in the obvious order.

The question is what happens if the process dies between them. And the sharper question underneath it: is there any arrangement of those two lines that makes the problem go away? Swap them, retry them, wrap them in a helper — try each in your head before reading on, because the interesting result is that none of them work, and the reason why is the whole idea.

Write the row first, then publish. If the process dies in between, the order exists and nobody is told. Silent loss — the worst kind, because the database looks correct.

So publish first, then write. Now a crash leaves a message announcing an order that does not exist. Downstream services bill a customer for nothing. Worse in most businesses, and no better in principle.

Try retrying the publish. That helps with a broker that is briefly unavailable, but it cannot help with the crash: the retry loop dies with the process.

Notice the pattern. Every arrangement fails in the same place: there are two systems, and no single act that changes both. The database can make many of its own writes atomic, and the broker can make its own delivery reliable, but neither has any authority over the other. This is the dual-write problem, and it is a property of the topology, not of the code.

What about a distributed transaction spanning both? XA two-phase commit does exist, and it trades one problem for another: the coordinator becomes a single point that, if it dies at the wrong moment, leaves participants holding locks and waiting for a decision. Most brokers support it poorly, and most teams judge the operational cost too high.

So try the other direction. Rather than making two systems agree, can we get down to one write? Ask what the message actually is at the moment of the order. It is not yet a delivery — it is an intention to deliver. And an intention is data. It can live in a table.

Follow that. In the same local transaction that inserts the order row, insert a second row into an outbox table holding the message body and its destination. One transaction, one database, ordinary atomicity that the database already provides. Either both rows exist or neither does. The crash that ruined every earlier arrangement now leaves a perfectly consistent state.

Then a separate process reads unsent rows from the outbox and publishes them, marking each one sent afterwards. But wait — publishing and marking are two writes to two systems again. Have we just moved the problem?

Almost, and this is the step worth being precise about. We have moved it somewhere it becomes survivable. If the relay publishes and dies before marking, it will find the row again on restart and publish a second time. That is a duplicate, not a loss. The dual-write problem has been converted from "the message may vanish" into "the message may arrive twice" — and duplicates are a problem a consumer can solve on its own, by ignoring a message ID it has already processed. Loss is not, because there is nothing left to notice.

That trade is the actual contribution of the pattern. Not exactly-once delivery, which this does not give you and which no broker really gives you either. What it gives you is at-least-once delivery with atomic capture, and the standing requirement that consumers be idempotent.

One more refinement. The relay can poll the outbox table on a timer, which is simple but adds latency and load. Or it can read the database's own replication log — change data capture, as tools like Debezium do — and turn committed outbox rows into messages without querying the table at all. The choice affects latency and operational shape; it does not change the guarantee, which came entirely from the single atomic write at the start.

Three clarifications.

First, the misconception: the outbox is often described as a way to get a transaction across a database and a broker. It is not. It gets you strictly out of needing one, by ensuring the only thing that must be atomic happens inside a single database.

Second, the outbox is a queue, and it inherits queue behaviour. If the relay stops, rows accumulate; if it never runs, the system is silently building a backlog while every write looks fine. Monitoring the age of the oldest unsent row is the health check that matters, and it is the one most often missing.

Third, ordering is not free. Two orders committed in sequence appear in the outbox in sequence, but a relay publishing them in parallel, or a broker with multiple partitions, can deliver them out of order. If order matters, it has to be arranged deliberately — a partition key, or a single-threaded relay per stream — and the outbox does not supply it by itself.

• Change data capture as a relay, and what it demands of database configuration.
• Idempotent consumers: deduplication keys, and how long a consumer must remember them.
• The inbox pattern, which applies the same reasoning to the receiving side.

Every term the collection defines is gathered in the glossary.