Poison messages

Why does one message that can never be processed stop every message behind it?

A queue holds forty thousand messages. One of them — message seven — has a malformed field that makes the consumer throw every single time it is touched. Nothing about messages eight through forty thousand is wrong. They are perfectly processable. And yet nothing moves.

Why should one bad item stop good ones? A queue is not a physical pipe. There is no bead stuck in a tube. So what exactly is the obstruction made of?

Start with what the consumer is doing, step by step. It takes a message. It works on it. It tells the broker it succeeded, and only then does the broker consider that message finished. If the consumer instead fails — or crashes, or simply says nothing for long enough — the broker must assume the work was not done and make the message available again. That redelivery is not a bug; it is the entire reason the queue is reliable. Without it, a consumer dying mid-work would silently lose the message.

Now put those two behaviours together. The consumer takes message seven, throws, and does not acknowledge. The broker redelivers message seven. The consumer takes it, throws, does not acknowledge. Do you see that we have built a perfect loop? Nothing in the mechanism has any notion of "this will never work". The broker's model has only two outcomes: done, or not yet done. A message that can never be done is indistinguishable, to the broker, from a message that has not been done yet.

That explains the loop. But it does not yet explain the blockage of the other forty thousand — and here the answer genuinely depends on the queue, which is worth being careful about rather than stating as one universal rule.

If the queue preserves order within some grouping — a partition in Kafka, a message group in an ordered queue, a single-consumer FIFO — then the blockage is total and inevitable. Order is the promise. The consumer cannot process message eight ahead of message seven without breaking it. So the whole partition halts at seven. This is head-of-line blocking, and it is not a flaw in the implementation; it is the price of the ordering guarantee, paid at the worst possible moment.

If the queue does not promise order, the picture is subtler. Other consumers can and do pick up message eight, so throughput does not fall to zero. But the poison message is still consuming something real: a delivery attempt, a consumer's time, and often a visibility timeout during which nothing else may claim it. One such message is negligible. A thousand of them — and a bad deploy produces a thousand in a minute — can occupy every consumer thread in a tight failure loop, and now the healthy messages starve too. So the blockage arrives by a different route, through capacity rather than ordering, but it arrives.

So what is missing from the mechanism? The broker needs a third outcome. Not "done" and not "not yet", but give up on this one. That is what a dead-letter queue provides: after some number of delivery attempts, the broker stops redelivering and moves the message to a separate queue instead. The main queue is unblocked; the message is not lost; a human or a repair process can examine it later.

Notice how modest that mechanism is. It diagnoses nothing. It has no idea whether the failure was permanent or a transient database blip that happened to occur five times. It counts. That crudeness is the real design tension: retries exist because most failures are transient, dead-lettering exists because some are not, and the counter is all that distinguishes them. Set the limit too low and you discard messages that would have succeeded next attempt; too high and a poison message burns capacity for a long time first.

Better systems do not rely on the counter alone. If the consumer can tell a failure is permanent — the payload does not parse, a required field is absent — it should say so and route the message straight to the dead-letter queue on the first attempt, retrying only what it believes to be transient. That classification is more work than setting a maximum, and it is the difference between a queue that recovers in seconds and one that thrashes for minutes.

Three refinements.

First, the misconception worth correcting: that the message is somehow blocking the queue physically, or that the broker is stuck. The broker is fine and is working exactly as designed. What is stuck is progress, and it is stuck because the retry rule has no termination condition. The obstruction is made of policy, not of matter.

Second, ordering guarantees and poison messages are two faces of the same trade. The stronger the ordering promise, the more completely one undeliverable message halts everything behind it. That is a reason to ask whether a stream genuinely needs total ordering, or ordering only within a customer, an account, an entity — because narrowing the ordering key narrows the blast radius of a poison message to exactly one key's worth of traffic.

Third, dead-lettering is not resolution. A dead-letter queue that nobody monitors is a place where data goes to be forgotten quietly, which is worse than failing loudly. The mechanism buys the live system its throughput back; it does not answer what should happen to the message, and that answer still has to be someone's job.

• How to choose an ordering key so a poison message stalls one customer's traffic rather than everyone's.
• Whether a consumer can reliably classify failures as permanent or transient, and what to do when it cannot.
• What a healthy operational practice around dead-letter queues looks like, including replay after a fix.

Every term the collection defines is gathered in the glossary.