Postmortem: Six-Hour Outage

Concepts covered: Retry & Failure Handling

This postmortem follows the order pipeline on a day before any of this lesson's patterns existed. Each decision behind it was ordinary when made; the outage comes from how they lined up. Read each event and name the pattern that would have stopped it before checking the table that follows.

The timeline

TimeEvent
09:00A tax service deploy adds a slow query; 30% of calls now take 1.5 s
09:04Calls past the 1 s timeout are retried by the client and the task, 4 attempts each: up to 16 calls per order
09:12Load reaches about 32,000 requests per second; every call times out
09:12Calls that exhaust retries are labeled permanent; every order goes to the DLQ
09:58The unbounded prefetch buffer runs out of memory; 4 restarts by 11:40 rewrite 1.9 million rows to the DLQ a second time
11:30Finance's hourly report shows almost no orders since 09:12: the first human signal
12:05On-call traces the timeouts to the tax service
13:00The tax team rolls back, but the retry storm keeps the service saturated
14:45Someone pauses the pipeline by hand and the tax service recovers
15:00The pipeline resumes and the outage is declared over, with 31 million rows in the DLQ
15:30An engineer copies every DLQ row onto the input log at full speed; the tax service saturates again
15:50The copy stops; a script replays at 600 per second for a little over 14 hours

Each event as a missing pattern

TimeMissing patternPreventionSection
09:04Retries at 2 layersRetries at a single layer, inside a budgetCascading Failure
09:12Mass timeouts handled per recordFailure rate fed to the classifier opens the breakerClassification
09:12DLQ went from 0.01% to 100% unseenAlert on DLQ rate by failure classDLQ as Signal
09:58Unbounded bufferCapped buffer pushing back to the logCascading Failure
11:30Consumers found it firstPaging on DLQ rate and table freshnessDLQ as Signal
13:00Nothing gave the service room to recoverBreaker with half-open probesCascading Failure
15:30Replay shared the live pathSeparate replay worker with its own limitReprocessing
15:50Duplicate DLQ rows, 1.9 millionUpsert on order_id guarded by event_seqReprocessing

The 09:12 misclassification matters most. A classifier reading the failure rate would have seen timeouts on more than half of calls within seconds, opened the breaker and left every order in the log. The DLQ flood, the duplicate rows and both replay mistakes disappear, and the 13:00 rollback ends the outage, because nothing is sending extra load to the service. What remains is a backlog in the log that drains once the tax service recovers.

The checklist

A clickstream pipeline enriches events with a geo-lookup service. During a geo-lookup slowdown, its DLQ grows by 2,000 rows per second, and the team plans to replay them afterward. Which change fixes the cause?

The checklist also has to hold up when the recovery tooling itself fails. At 09:58 in the postmortem, 4 restarts wrote 1.9 million orders to the DLQ twice, and a replay worker can crash halfway through its selection in the same way. A design passes the replay items only if running the same selection a second time leaves every row of curated.orders unchanged.

A team replays 6 million DLQ rows with a worker throttled to the tax service's headroom. Halfway through, the worker crashes and restarts with the same selection. What keeps curated.orders correct?

About this lesson

This section is part of the Failure Handling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.