Dead Letter Queue Basics
Concepts covered: Dead Letter Queues
A bad message blocks a whole partition
A consumer reads a queue partition in order and commits its offset only after processing a message. Suppose offset 918223 carries the amount 12,50, which can never parse. The consumer cannot commit past it, so every later message on that partition waits. This is head-of-line blocking: a single permanent failure halts traffic with nothing wrong with it, and a retry loop with no exit keeps the consumer on that message indefinitely.
A dead letter queue (DLQ) is a second destination for messages the consumer has classified as unprocessable, each stored with the reason it failed. After writing the message to the DLQ, the consumer commits past it and moves on. The bad record is kept for a person to inspect, and the good records behind it load within seconds instead of waiting days for a fix.
- trip_payments (queue, message queue)
- payments_loader (transform, consumer); on error: permanent: write to DLQ, commit offset
- fact_payments (storage, warehouse table)
- trip_payments_dlq (queue, message queue); alerting: arrivals and oldest-entry age
- trip_payments -> payments_loader
- payments_loader -> fact_payments
- payments_loader -> trip_payments_dlq
What goes in, and what stays out
- Send to the DLQ
- A payload that fails to parse, such as an amount of 12,50
- A value that breaks a rule no retry can change, like a negative fare
- A schema version this consumer has never seen
- A message that crashed the consumer on 5 deliveries in a row
- Keep out of the DLQ
- A timeout from the warehouse
- A 503 from the currency service
- A dropped connection
- A rate-limit response that says when to come back
The left column holds failures that belong to the message: replayed unchanged, it fails again. The right column holds failures of the world around it. Routing those to the DLQ during a 20-minute warehouse outage would bury thousands of valid payments among the broken ones, each needing a manual replay. They get retried instead, and a message whose retries run out stays uncommitted in the main queue to be tried again.
The last item on the left needs care. A poison message crashes the process, for example by exhausting memory, so no error handler ever runs: the consumer restarts, reads the same offset and crashes again. The fix is to count deliveries somewhere a crash can't erase, with one counter per partition and offset. The consumer increments it before handling a message and, at 5 deliveries, sends the message to the DLQ unprocessed.
Which failure should the loader send to trip_payments_dlq?
- The warehouse answers every write with a 503 for 40 seconds
- A payment whose fare is -14.00, which the rules forbid
- The warehouse times out on 3 writes in a row
- The consumer loses its connection during a network change
The envelope
An entry holding only error text is useless a week later. Each DLQ entry wraps the original message in an envelope. Next to the payload it records the source offset and the error, plus the version of the code that rejected it.
{
"original_payload": "{\"payment_id\": \"p-7003\", \"amount\": \"12,50\", \"currency\": \"EUR\"}",
"source": {
"topic": "trip_payments",
"partition": 4,
"offset": 918223
},
"error": {
"class": "permanent",
"type": "AmountParseError",
"message": "amount '12,50' is not a decimal"
},
"attempts": 1,
"first_failed_at": "2026-10-23T18:00:04Z",
"last_failed_at": "2026-10-23T18:00:04Z",
"consumer_version": "payments-loader 2.14.0",
"schema_version": 7
}| Field | Why replay needs it |
|---|---|
| original_payload | Replay sends these exact bytes back through the loader; a re-serialized copy can hide the bug |
| source topic, partition, offset | Locates the message in the source and shows whether it was ever committed |
| error class and type | Groups 4,000 entries into 3 causes instead of 4,000 stack traces |
| consumer_version | Tells you whether a fix has shipped since, so a replay is worth trying |
Retention longer than the source
The envelope carries a copy of the payload because the source won't keep one. If trip_payments retains messages for 7 days, the original of offset 918223 is deleted on day 8, and from then on the DLQ holds the only copy of that payment. DLQ retention therefore has to cover the slowest realistic path from failure to replay.
| Step | Slow but realistic |
|---|---|
| Entry lands Friday 18:00, alert read Monday morning | 3 days |
| Root cause found, fix reviewed and deployed | 4 days |
| Replay run and row counts checked | 1 day |
| Total from failure to replay | 8 days |
A 7-day DLQ would delete this payment a day before the replay. Set DLQ retention well past the worst case, such as 30 days.
- Do
- Alert on DLQ arrivals and on the age of the oldest entry
- Name an owner: the team that owns the consumer
- Replay through the same loader code, so validation runs again
- Avoid
- Leave the DLQ unwatched: a queue nobody reads is a delete with a delay
- Edit payloads in place; publish a corrected copy and keep the original
DLQ entries hold only an error string and a timestamp. The decimal-comma fix ships 12 days after the first entry; trip_payments keeps messages 7 days. What happens to the first day's failed payments?
- They replay from the DLQ through the fixed loader
- They are re-read from the topic using the failure timestamps
- They are lost: no payload in the DLQ, and the topic copy expired
- They load on their own once the fixed loader restarts
About this lesson
This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.