AdvancedPipeline Architecture · 38 min · 12 challenges

Failure Handling: Advanced

Uber's engineers have written about sending failed payment events for the Driver Injury Protection program through delayed retry queues and, as a last stop, a dead letter queue. When a premium-calculation service slows to a crawl on a Friday evening, decisions made long before that evening determine whether those queues protect drivers' coverage or bury millions of good payments.

Failure Classification by Design

Daily Life
Interviews
Splitting failures into transient and permanent answers a single question: retry or not. A production pipeline has to answer 4 questions for every failure. Should it retry? Should it stop calling the dependency? Where does the failed record go? Does someone get woken up? The answers to all 4 depend on 2 properties of the failure: how long it will last, and how much of the traffic it hits.

Every example in this lesson uses the same pipeline. It reads 2,000 order events per second from a durable log, calls a tax service that can serve 3,000 requests per second, and writes each priced order to curated.orders, keyed on order_id.

Duration and scope

Scope is the newer idea. A record-scoped failure hits a single record because of something inside it: a malformed amount, a missing customer ID. A systemic failure hits every record because of something outside them all: an overloaded service, an expired credential, a bug in freshly deployed code. The same error can land on either side. A timeout on 1 call in 10,000 is record-scoped noise, while timeouts on every call are systemic.
ClassDurationScopeExampleResponse
Transient recordSecondsSingle recordA lone 503 or connection resetRetry in place with backoff and jitter
Transient systemicMinutes to hoursAll recordsTax service overloaded, 429 on every callOpen the breaker, leave records in the source log
Permanent recordForeverSingle record`tip_amount` holds the string 'abc'Write it to the DLQ and keep going
Permanent systemicUntil a human actsAll recordsExpired credential returning 401; a bug in the transformHalt the pipeline and page the owner
The bottom row is where pipelines most often go wrong. Record by record, a permanent systemic failure looks like a permanent record failure: the call returns 401 and the order cannot be priced. A pipeline that checks only the error type sends every order to the DLQ, and an hour later the DLQ holds 7.2 million good orders. All of them need the same fix, a new credential, and none of them needed a decision of its own.

The classifier is code, and it reads the failure rate

Classification belongs in a single function that every retry, breaker and DLQ decision calls, so the rules live in one place and can be tested. It takes 2 inputs: the error, and the share of recent records that failed with the same kind of error. For timeouts and overload responses, that share tracks the tax service. For parse errors, it tracks the payloads. That share is what promotes a record-scoped verdict to a systemic one.
from dataclasses import dataclass
TRANSIENT_STATUS = {429, 502, 503, 504}
AUTH_STATUS = {401, 403}
SYSTEMIC_RATE = 0.5
@dataclass
class Verdict:
failure_class: str
action: str
def classify(error, same_kind_failure_rate):
status = getattr(error, "status", None)
systemic = same_kind_failure_rate >= SYSTEMIC_RATE
if isinstance(error, DuplicateKeyError):
return Verdict("already_applied", "ack")
if status in AUTH_STATUS:
return Verdict("permanent_systemic", "halt_and_page")
if status in TRANSIENT_STATUS or isinstance(error, (TimeoutError, ConnectionError)):
if systemic:
return Verdict("transient_systemic", "open_breaker")
return Verdict("transient_record", "retry_with_backoff")
if isinstance(error, (ValueError, KeyError)):
if systemic:
return Verdict("permanent_systemic", "halt_and_page")
return Verdict("permanent_record", "send_to_dlq")
return Verdict("unknown", "retry_then_dlq")

A duplicate-key error on an insert-if-absent write keyed on order_id and event_seq together means an earlier attempt already landed this exact event, so the record counts as a success and is acknowledged. A ValueError on half of all records points to a deployed bug or a producer format change, so the per-record verdict is overridden and the pipeline halts. An error the function has never seen gets a short retry and then the DLQ, and the count of unknown verdicts is tracked: when it climbs past a handful per hour, the classifier needs a new rule.

One table drives retry, breaker, queue and runbook

With classes defined, every mechanism downstream reads from the same table. That includes the runbook, the page an on-call engineer opens when an alert fires. It gets an entry per class instead of an entry per error message, which keeps it short enough to use at 3 a.m.
ClassRetryBreakerRecord goes toPagesRunbook first step
Transient recordUp to 4 attempts, exponential backoff, jitterCounts toward failure rateDLQ after attempt 4NoNone
Transient systemicPaused while breaker is openOpens at 50% failures over 30 sStays in the source logAfter 10 min openCheck the dependency's health and recent deploys
Permanent recordNoneNot countedDLQ with error signatureNoGroup the DLQ by signature
Permanent systemicNoneAll calls stoppedStays in the source logImmediatelyCheck the last deploy and credential expiry
Already appliedNoneNot countedAcknowledgedNoNone
TIP
Write the class table before any retry code. An argument about retry counts or DLQ routing then turns into a question of which row a failure belongs to, and that question has a testable answer.
Multiple Choice

A credential rotation at the tax service goes wrong, and from 14:00 every call from the order pipeline returns 401. Today the pipeline sends any record that fails 4 attempts to the DLQ. What should it do instead?

The DLQ as a Quality Signal

Daily Life
Interviews
Consumers notice bad data when a dashboard number looks wrong, often hours after the cause. The DLQ sees the cause within seconds, because every record a producer change breaks lands there the moment it is processed. With the right metrics, the DLQ becomes the earliest warning the pipeline has about the systems that feed it.

What a DLQ row has to carry

A metric can slice only by a field the DLQ row stores. A DLQ holding the payload and a raw error message answers how many records failed. It can't say which producer release broke which field, and that is the question that finds a regression.
ops.orders_dlq
dlq_idPKbigint
order_idbigint
event_seqbigint
payloadvarchar
producer_versionvarchar
failure_classvarchar
error_signaturevarchar
error_messagevarchar
attempt_countinteger
first_failed_attimestamp
statusvarchar
ticketvarchar
ops.orders_throughput
hour_startPKtimestamp
producer_versionPKvarchar
records_inbigint
records_writtenbigint
records_dlqbigint
records_filteredbigint

The error_signature column is the raw message with its values stripped out. cannot cast 'abc' to decimal at tip_amount and cannot cast '12,50' to decimal at tip_amount both become cannot cast <v> to decimal at tip_amount. Grouping by raw message produces a group per bad value; grouping by signature produces a group per bug.

Rate by slice, against a baseline

The order pipeline's normal DLQ rate is 0.01%, about 720 rows an hour. At 10:00 the order service starts a canary of release v4.12, a trial rollout on 5% of its hosts. The release sends tip_amount as a string whenever a tip was edited after checkout, which happens on 15% of orders. The next hour looks like this:

producer_versionrecords_indlq_rowsdlq_rate
v4.116,840,0006840.01%
v4.12360,00054,00015.00%
All versions7,200,00054,6840.76%
Consider which alerts that hour trips. An alert on 100,000 DLQ rows per hour stays silent. A pipeline-wide rate alert at 1% stays silent too, since 0.76% sits under it. Sliced by producer version, v4.12 fails at 15%, a figure no healthy release produces. That alert fires during the canary and the rollout stops at 5% of hosts. At full rollout, the same bug would send 1.08 million orders an hour to the DLQ.
1,500x
v4.12 DLQ rate over the baseline
WITH dlq AS (
SELECT
producer_version,
COUNT(*) AS dlq_rows
FROM ops.orders_dlq
WHERE first_failed_at >= DATE_TRUNC('hour', CURRENT_TIMESTAMP) - INTERVAL '1' HOUR
AND first_failed_at < DATE_TRUNC('hour', CURRENT_TIMESTAMP)
GROUP BY producer_version
),
traffic AS (
SELECT
producer_version,
SUM(records_in) AS records_in
FROM ops.orders_throughput
WHERE hour_start = DATE_TRUNC('hour', CURRENT_TIMESTAMP) - INTERVAL '1' HOUR
GROUP BY producer_version
)
SELECT
t.producer_version,
t.records_in,
COALESCE(d.dlq_rows, 0) AS dlq_rows,
ROUND(
COALESCE(d.dlq_rows, 0) * 100.0 / NULLIF(t.records_in, 0),
2
) AS dlq_pct
FROM traffic AS t
LEFT JOIN dlq AS d
ON d.producer_version = t.producer_version
ORDER BY dlq_pct DESC

The 4 DLQ metrics worth alerting on

  • Rate by slice: dlq_rows / records_in per producer version and failure class. Catches a regression while a canary still touches a small share of traffic.
  • New signature: an error_signature absent for the past 14 days. A new bug shows up as a new group even at low volume.
  • Oldest untriaged row: the age of the oldest row with status = 'new'. A DLQ nobody reads loses data slowly, and this age says how slowly.
  • Conservation: records_in = records_written + records_dlq + records_filtered. A gap means records vanished without reaching the DLQ.

The conservation check exists because a quiet DLQ proves nothing. If a code change wraps the parse step in a try block that logs the error and moves on, failures stop reaching the DLQ and its rate falls to zero, which looks like an improvement. Conservation exposes it: 7,200,000 records in against 7,180,000 written, 0 in the DLQ and 0 filtered leaves 20,000 orders unaccounted for.

Multiple Choice

Right after a pipeline deploy, the order pipeline's DLQ rate drops from 0.01% to 0.00%, while records_in holds steady at 7.2 million per hour. What should an engineer check first?

Reprocessing From the DLQ

Daily Life
Interviews
A DLQ that only collects records loses them more slowly. Recovery needs a path back: find the broken records, write down what is known about them, fix the cause, and send them through the pipeline again without starving live traffic or corrupting the curated table.

A lifecycle for every DLQ row

statusSet byMeaning
newThe pipelineFailed, and nobody has looked yet
triagedOn-call engineerThe signature has an owner and a ticket
fix_deployedOwnerThe fix is live; rows are safe to replay
replayedReplay workerWritten to curated.orders
refailedReplay workerFailed again; attempt_count raised
discardedOwner, with a reasonNever to be replayed, such as test traffic

Annotation works a signature at a time. A single update marks all 54,000 tip_amount rows from the v4.12 canary as triaged with a ticket number. The status column lets replay select only rows whose cause is fixed, and it ends loops: a row that reaches refailed a third time stays put until a human looks.

Replay through a separate, throttled path

The quickest-looking replay copies DLQ rows back onto the pipeline's input log. Those old records then queue behind live ones and draw on the same tax capacity live orders need. A dedicated replay worker runs the same transform code under its own rate limit, so replay cannot take capacity from live orders as long as that limit fits the dependency's current headroom.
Set that limit from the dependency's headroom. Live traffic uses 2,000 of the tax service's 3,000 requests per second. Holding 400 back for bursts leaves replay 600 per second. At that rate the 54,000 canary rows replay in 90 seconds, and a backlog of 2.1 million rows takes roughly 58 minutes.
Durable log
Order log
72 h retention
Same transform code
Replay worker
PARALLELISM4 workersIDEMPOTENCYevent seq-guarded upsert
Tax enrichment
Live transform
CONSUMERS32ERRORclassify, then retry, break or DLQ
3
Tax service
000 req/s capacity
DLQ table
orders dlq
MONITORDLQ rate by producer_version
Keyed on order_id
curated.orders
IDEMPOTENCYupsert on order id

Live and replay paths run the same transform code and share the tax service. Replay reads fixed rows from the DLQ under its own limit of 600 requests per second, upserts them into curated.orders, and writes rows that fail again back to the DLQ.

Idempotent writes and the stale replay

Replay has to be safe to run twice, because the replay worker can crash halfway like any process. An upsert keyed on order_id covers that: a second run rewrites the same row. Replay also meets a hazard live traffic rarely does, a DLQ row older than what the table already holds. Order 88123 failed at event_seq 4, when its tip was edited. Its seq 5, a cancellation, succeeded an hour later. A plain upsert of seq 4 would un-cancel the order.

The guard is a condition on the write: apply a replayed row only when its event_seq is greater than the stored one. Running the same condition as a read-only dry run shows what replay will do before it touches the table.

SELECT
d.order_id,
d.event_seq AS replay_seq,
o.event_seq AS current_seq,
CASE
WHEN o.order_id IS NULL THEN 'insert'
WHEN d.event_seq > o.event_seq THEN 'update'
ELSE 'skip_stale'
END AS replay_action
FROM ops.orders_dlq AS d
LEFT JOIN curated.orders AS o
ON o.order_id = d.order_id
WHERE d.status = 'fix_deployed'
AND d.error_signature = 'cannot cast <v> to decimal at tip_amount'
ORDER BY d.order_id
order_idreplay_seqcurrent_seqreplay_action
881211NULLinsert
8812242update
8812345skip_stale

A dry run with 2% of rows in skip_stale is normal: those orders moved on while their failed event waited. A dry run with 60% in skip_stale means the selection is wrong, often rows a crashed replay wrote without updating their status. The guard turns that second pass into skips instead of overwrites.

Multiple Choice

A replay of 2.1 million DLQ rows is planned for peak hours. The tax service serves 3,000 requests per second, live traffic uses 2,500, and 400 stay in reserve for bursts. About how long will replay take?

Cascading Failure and Backpressure

Daily Life
Interviews
A cascade starts with a single slow dependency and ends with failures in systems that were never broken. Underneath it is a queue that fills: whenever records arrive faster than a stage can finish them, the difference piles up somewhere, and every somewhere has a size limit.

Arrival rate minus service rate

Suppose a slow database query drops the tax service's capacity from 3,000 requests per second to 1,200. Orders still arrive at 2,000 per second. The order pipeline's in-memory buffer has no size limit, and its process has memory for around 500,000 records, so the buffer grows by 800 records every second until that memory runs out.
Minutes since slowdownBuffered recordsProcess memory
00Empty
296,00019% full
5240,00048% full
10480,00096% full
10.4500,000Full: the process runs out of memory
The crash makes the next minute worse. The restarted process re-reads everything since its last committed offset, so it starts with a backlog and calls the slow service harder than before it died.

Latency drains pools and retries multiply load

Little's law says the number of requests in flight equals arrival rate times latency. At 2,000 requests per second and 50 ms each, the pipeline holds 100 calls open. When latency climbs to 2 seconds, it needs 4,000. A connection pool of 200 runs dry, and records time out while waiting for a connection they never got.

Retries stack on top. If the HTTP client makes 4 attempts and the task wrapping it also makes 4, a single order can produce 16 calls. At 2,000 orders per second, a struggling tax service can face 32,000 requests per second, more than 10 times its healthy capacity. That load keeps it down even after its own fix ships.

3 ways to bound the damage

TacticWhat it doesWhat it costsUse when
BackpressureA full buffer stops the consumer pulling; records wait in the durable logLag grows; nothing is lost while lag stays within retentionThe source is durable and the slowdown ends within retention
Load sheddingLow-priority records are deferred or dropped so critical ones keep flowingDeferred work runs later; dropped work is goneSome consumers can wait and others cannot
Degraded fallbackA cached or estimated answer replaces the slow call, flagged as estimatedResults stay approximate until reprocessedAn approximate answer now beats an exact one later
Backpressure holds only if every stage between the slow service and the durable source honors it. A stage with an unbounded buffer absorbs the pressure, hides it from the stages before it, and becomes the stage that runs out of memory. The order log keeps 72 hours of events, so through a full tax outage, backpressure loses nothing for the first 72 hours. From hour 72 onward, each further hour of outage expires the oldest hour of unread orders.
Shedding needs a priority on every record, set at design time. Payment events feed the ledger and cannot wait; cart-edit events feed only a marketing attribution table and can wait a day. Under slowdown, the pipeline writes cart edits to a deferred table and spends the tax service's 1,200 requests per second on payment events first.
Classification picks the tactic. Timeouts on every call are transient systemic, so the breaker opens and backpressure keeps records in the log. Routing those timeouts to the DLQ would turn a slowdown into millions of good orders awaiting replay.
✓Do
  • Cap every in-memory buffer and stop pulling when it is full
  • Retry at a single layer, inside a retry budget
  • Tag each record with a priority before the pipeline needs to shed
  • Alert when source lag passes half of the log's retention
✗Don't
  • Let any stage buffer without a size limit
  • Stack client, task and orchestrator retries on the same call
  • Send records to the DLQ because a dependency was slow
  • Shed records without logging what was dropped
Multiple Choice

The tax service goes fully down. The breaker opens, every buffer is capped, and backpressure holds orders in the log, which keeps 72 hours of events. The outage lasts 80 hours. What happens to the orders?

Postmortem: Six-Hour Outage

Daily Life
Interviews
This postmortem follows the order pipeline on a day before any of this lesson's patterns existed. Each decision behind it was ordinary when made; the outage comes from how they lined up. Read each event and name the pattern that would have stopped it before checking the table that follows.

The timeline

TimeEvent
09:00A tax service deploy adds a slow query; 30% of calls now take 1.5 s
09:04Calls past the 1 s timeout are retried by the client and the task, 4 attempts each: up to 16 calls per order
09:12Load reaches about 32,000 requests per second; every call times out
09:12Calls that exhaust retries are labeled permanent; every order goes to the DLQ
09:58The unbounded prefetch buffer runs out of memory; 4 restarts by 11:40 rewrite 1.9 million rows to the DLQ a second time
11:30Finance's hourly report shows almost no orders since 09:12: the first human signal
12:05On-call traces the timeouts to the tax service
13:00The tax team rolls back, but the retry storm keeps the service saturated
14:45Someone pauses the pipeline by hand and the tax service recovers
15:00The pipeline resumes and the outage is declared over, with 31 million rows in the DLQ
15:30An engineer copies every DLQ row onto the input log at full speed; the tax service saturates again
15:50The copy stops; a script replays at 600 per second for a little over 14 hours
2 h 30 min
until a finance report, not an alert

Each event as a missing pattern

TimeMissing patternPreventionSection
09:04Retries at 2 layersRetries at a single layer, inside a budgetCascading Failure
09:12Mass timeouts handled per recordFailure rate fed to the classifier opens the breakerClassification
09:12DLQ went from 0.01% to 100% unseenAlert on DLQ rate by failure classDLQ as Signal
09:58Unbounded bufferCapped buffer pushing back to the logCascading Failure
11:30Consumers found it firstPaging on DLQ rate and table freshnessDLQ as Signal
13:00Nothing gave the service room to recoverBreaker with half-open probesCascading Failure
15:30Replay shared the live pathSeparate replay worker with its own limitReprocessing
15:50Duplicate DLQ rows, 1.9 millionUpsert on order_id guarded by event_seqReprocessing
The 09:12 misclassification matters most. A classifier reading the failure rate would have seen timeouts on more than half of calls within seconds, opened the breaker and left every order in the log. The DLQ flood, the duplicate rows and both replay mistakes disappear, and the 13:00 rollback ends the outage, because nothing is sending extra load to the service. What remains is a backlog in the log that drains once the tax service recovers.

The checklist

Every failure gets a class from duration and scope, and the classifier reads the recent failure rate
Every failure gets a class from duration and scope, and the classifier reads the recent failure rate
Retries live at a single layer, with exponential backoff, jitter and a retry budget
Retries live at a single layer, with exponential backoff, jitter and a retry budget
Each dependency call sits behind a circuit breaker with half-open probes
Each dependency call sits behind a circuit breaker with half-open probes
Every buffer is capped, and a full buffer stops the consumer from pulling
Every buffer is capped, and a full buffer stops the consumer from pulling
Records carry a priority, so the pipeline can defer or shed them under slowdown
Records carry a priority, so the pipeline can defer or shed them under slowdown
DLQ rows store producer version, failure class and error signature
DLQ rows store producer version, failure class and error signature
Alerts cover DLQ rate by slice, new signatures, the oldest untriaged row and conservation
Alerts cover DLQ rate by slice, new signatures, the oldest untriaged row and conservation
Replay runs on its own throttled path and writes with an event_seq-guarded upsert
Replay runs on its own throttled path and writes with an event_seq-guarded upsert
The runbook has an entry per failure class
The runbook has an entry per failure class
TIP
Run the checklist against a design before it ships: name a failure from each class and trace where each record ends up. A record that can end up nowhere, or in 2 places, marks the gap.
Multiple Choice

A clickstream pipeline enriches events with a geo-lookup service. During a geo-lookup slowdown, its DLQ grows by 2,000 rows per second, and the team plans to replay them afterward. Which change fixes the cause?

The checklist also has to hold up when the recovery tooling itself fails. At 09:58 in the postmortem, 4 restarts wrote 1.9 million orders to the DLQ twice, and a replay worker can crash halfway through its selection in the same way. A design passes the replay items only if running the same selection a second time leaves every row of curated.orders unchanged.

Multiple Choice

A team replays 6 million DLQ rows with a worker throttled to the tax service's headroom. Halfway through, the worker crashes and restarts with the same selection. What keeps curated.orders correct?

PUTTING IT ALL TOGETHER

> Uber has published how payments for its Driver Injury Protection program move through retry queues with growing delays before landing in a dead letter queue. Take that design to the night a premium-calculation service slows down after a deploy, and walk each failure through it.

Classification: timeouts on most calls mark the slowdown as transient systemic, so the breaker opens and payments wait in the source log instead of draining through the retry queues.
Signal: a DLQ rate split by producer version shows whether the slowdown or a bad producer release causes the failures, minutes before drivers or finance notice.
Backpressure: capped buffers hold payments in the log, and the log's retention sets how long the slowdown can last before anything is lost.
Replay: once the service recovers, a throttled worker drains the DLQ within the service's headroom, and an upsert on payment ID makes a crashed replay safe to rerun.
Postmortem: if the night still goes wrong, each event maps to a missing checklist item, which turns the incident into specific design changes.
KEY TAKEAWAYS
Classify every failure by duration and scope, and let the classifier read the recent failure rate so mass failures count as systemic.
A permanent systemic failure belongs in a halted pipeline with a page, and its records stay in the source log.
Slice the DLQ rate by producer version and failure class so a canary regression fires an alert before full rollout.
Pair DLQ metrics with a conservation check, because a silent DLQ can hide swallowed failures.
Replay on a separate path throttled to the dependency's headroom, writing with an upsert guarded by event sequence.
Cascades are queues filling: cap buffers, push back to the durable log, shed by priority, and retry at a single layer.

Classify failures by duration and scope, read the DLQ as a warning, replay safely, stop cascades.

Category
Pipeline Architecture
Difficulty
advanced
Duration
38 minutes
Challenges
12 hands-on challenges

Topics covered: Failure Classification by Design, The DLQ as a Quality Signal, Reprocessing From the DLQ, Cascading Failure and Backpressure, Postmortem: Six-Hour Outage

Lesson Sections

  1. Failure Classification by Design (concepts: Retry & Failure Handling)

    Splitting failures into transient and permanent answers a single question: retry or not. A production pipeline has to answer 4 questions for every failure. Should it retry? Should it stop calling the dependency? Where does the failed record go? Does someone get woken up? The answers to all 4 depend on 2 properties of the failure: how long it will last, and how much of the traffic it hits. Duration and scope Scope is the newer idea. A record-scoped failure hits a single record because of somethin

  2. The DLQ as a Quality Signal

    Consumers notice bad data when a dashboard number looks wrong, often hours after the cause. The DLQ sees the cause within seconds, because every record a producer change breaks lands there the moment it is processed. With the right metrics, the DLQ becomes the earliest warning the pipeline has about the systems that feed it. What a DLQ row has to carry A metric can slice only by a field the DLQ row stores. A DLQ holding the payload and a raw error message answers how many records failed. It can'

  3. Reprocessing From the DLQ (concepts: Dead Letter Queues)

    A DLQ that only collects records loses them more slowly. Recovery needs a path back: find the broken records, write down what is known about them, fix the cause, and send them through the pipeline again without starving live traffic or corrupting the curated table. A lifecycle for every DLQ row Replay through a separate, throttled path The quickest-looking replay copies DLQ rows back onto the pipeline's input log. Those old records then queue behind live ones and draw on the same tax capacity li

  4. Cascading Failure and Backpressure

    A cascade starts with a single slow dependency and ends with failures in systems that were never broken. Underneath it is a queue that fills: whenever records arrive faster than a stage can finish them, the difference piles up somewhere, and every somewhere has a size limit. Arrival rate minus service rate Suppose a slow database query drops the tax service's capacity from 3,000 requests per second to 1,200. Orders still arrive at 2,000 per second. The order pipeline's in-memory buffer has no si

  5. Postmortem: Six-Hour Outage (concepts: Retry & Failure Handling)

    This postmortem follows the order pipeline on a day before any of this lesson's patterns existed. Each decision behind it was ordinary when made; the outage comes from how they lined up. Read each event and name the pattern that would have stopped it before checking the table that follows. The timeline Each event as a missing pattern The 09:12 misclassification matters most. A classifier reading the failure rate would have seen timeouts on more than half of calls within seconds, opened the break