Poison Messages
A message that always fails processing blocks or endlessly re-queues.
By the end of this lesson, isolate messages that can never succeed so they don't stall the pipeline.
A poison message is one the consumer can never successfully process — malformed payload, a reference to deleted data, a bug for that specific shape. If the consumer retries on failure, the poison message returns to the front of the queue forever, blocking everything behind it or spinning the consumer at 100% doing nothing.
Retry loop vs. dead-letter
- A consumer that keeps failing on a message loops; after N attempts the message is moved to a dead-letter queue for human inspection.
flowchart LR q[(Queue)] --> c[Consumer] c -->|fail| c c -->|after N tries| dlq[(Dead-letter queue)] dlq --> ops[Human / repair]
The standard fix is a dead-letter queue (DLQ): after N failed attempts, move the message aside so the pipeline continues, and alert humans to inspect it. Combine with a retry cap, structured error capture, and schema validation at the edge to reject bad messages before they enter.
Poison message
- Trigger
- A message that deterministically fails and is retried without a cap.
- Symptom
- A stuck or looping consumer; head-of-line blocking; wasted throughput.
- Blast radius
- The partition/queue it sits in and everything behind it.
- Mitigation
- Dead-letter queue after N retries, retry caps, edge validation, alerting.
Why cap retries and route to a DLQ instead of retrying forever?
A deterministically-failing message will never succeed; endless retries block the queue and waste capacity. A DLQ isolates it so the pipeline keeps moving.