Postmortem: Six-Hour Outage
Concepts covered: Retry & Failure Handling
This postmortem follows the order pipeline on a day before any of this lesson's patterns existed. Each decision behind it was ordinary when made; the outage comes from how they lined up. Read each event and name the pattern that would have stopped it before checking the table that follows.
The timeline
| Time | Event |
|---|---|
| 09:00 | A tax service deploy adds a slow query; 30% of calls now take 1.5 s |
| 09:04 | Calls past the 1 s timeout are retried by the client and the task, 4 attempts each: up to 16 calls per order |
| 09:12 | Load reaches about 32,000 requests per second; every call times out |
| 09:12 | Calls that exhaust retries are labeled permanent; every order goes to the DLQ |
| 09:58 | The unbounded prefetch buffer runs out of memory; 4 restarts by 11:40 rewrite 1.9 million rows to the DLQ a second time |
| 11:30 | Finance's hourly report shows almost no orders since 09:12: the first human signal |
| 12:05 | On-call traces the timeouts to the tax service |
| 13:00 | The tax team rolls back, but the retry storm keeps the service saturated |
| 14:45 | Someone pauses the pipeline by hand and the tax service recovers |
| 15:00 | The pipeline resumes and the outage is declared over, with 31 million rows in the DLQ |
| 15:30 | An engineer copies every DLQ row onto the input log at full speed; the tax service saturates again |
| 15:50 | The copy stops; a script replays at 600 per second for a little over 14 hours |
- 2 h 30 min
Each event as a missing pattern
| Time | Missing pattern | Prevention | Section |
|---|---|---|---|
| 09:04 | Retries at 2 layers | Retries at a single layer, inside a budget | Cascading Failure |
| 09:12 | Mass timeouts handled per record | Failure rate fed to the classifier opens the breaker | Classification |
| 09:12 | DLQ went from 0.01% to 100% unseen | Alert on DLQ rate by failure class | DLQ as Signal |
| 09:58 | Unbounded buffer | Capped buffer pushing back to the log | Cascading Failure |
| 11:30 | Consumers found it first | Paging on DLQ rate and table freshness | DLQ as Signal |
| 13:00 | Nothing gave the service room to recover | Breaker with half-open probes | Cascading Failure |
| 15:30 | Replay shared the live path | Separate replay worker with its own limit | Reprocessing |
| 15:50 | Duplicate DLQ rows, 1.9 million | Upsert on order_id guarded by event_seq | Reprocessing |
The 09:12 misclassification matters most. A classifier reading the failure rate would have seen timeouts on more than half of calls within seconds, opened the breaker and left every order in the log. The DLQ flood, the duplicate rows and both replay mistakes disappear, and the 13:00 rollback ends the outage, because nothing is sending extra load to the service. What remains is a backlog in the log that drains once the tax service recovers.
The checklist
- Every failure gets a class from duration and scope, and the classifier reads the recent failure rate
- Retries live at a single layer, with exponential backoff, jitter and a retry budget
- Each dependency call sits behind a circuit breaker with half-open probes
- Every buffer is capped, and a full buffer stops the consumer from pulling
- Records carry a priority, so the pipeline can defer or shed them under slowdown
- DLQ rows store producer version, failure class and error signature
- Alerts cover DLQ rate by slice, new signatures, the oldest untriaged row and conservation
- Replay runs on its own throttled path and writes with an event_seq-guarded upsert
- The runbook has an entry per failure class
A clickstream pipeline enriches events with a geo-lookup service. During a geo-lookup slowdown, its DLQ grows by 2,000 rows per second, and the team plans to replay them afterward. Which change fixes the cause?
- Raise the DLQ storage limit so it can hold a full day of failed events
- Add a third retry layer so slow calls get more chances before reaching the DLQ
- Feed the failure rate to the classifier so mass timeouts open the breaker and stay in the log
- Replay the DLQ continuously at full speed so it never holds more than a few minutes
The checklist also has to hold up when the recovery tooling itself fails. At 09:58 in the postmortem, 4 restarts wrote 1.9 million orders to the DLQ twice, and a replay worker can crash halfway through its selection in the same way. A design passes the replay items only if running the same selection a second time leaves every row of curated.orders unchanged.
A team replays 6 million DLQ rows with a worker throttled to the tax service's headroom. Halfway through, the worker crashes and restarts with the same selection. What keeps curated.orders correct?
- The upsert on order_id guarded by event_seq, which turns already-written rows into skips
- The throttle, which keeps replay slow enough that duplicate writes cannot occur
- The DLQ rate alert, which fires if the restarted worker writes a row a second time
- The circuit breaker, which opens when the same order_id is written a second time
About this lesson
This section is part of the Failure Handling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.