Failure Handling: Intermediate
Friday, 18:00, at Uber. A fare-adjustment event arrives with its amount written as 12,50, a German-locale decimal. The payments consumer can't parse it. Because it retries the event forever, every trip receipt behind it stalls. At 18:10 the currency service stops answering and 200 workers retry in lockstep. Whether dinner rides get paid tonight depends on where each failure is sent.
A bad message blocks a whole partition
A consumer reads a queue partition in order and commits its offset only after processing a message. Suppose offset 918223 carries the amount 12,50, which can never parse. The consumer cannot commit past it, so every later message on that partition waits. This is head-of-line blocking: a single permanent failure halts traffic with nothing wrong with it, and a retry loop with no exit keeps the consumer on that message indefinitely.
Parsed payments flow to fact_payments. A message the loader classifies as permanent goes to trip_payments_dlq inside an envelope, and the loader commits past it.
What goes in, and what stays out
- A payload that fails to parse, such as an amount of 12,50
- A value that breaks a rule no retry can change, like a negative fare
- A schema version this consumer has never seen
- A message that crashed the consumer on 5 deliveries in a row
- A timeout from the warehouse
- A 503 from the currency service
- A dropped connection
- A rate-limit response that says when to come back
The last item on the left needs care. A poison message crashes the process, for example by exhausting memory, so no error handler ever runs: the consumer restarts, reads the same offset and crashes again. The fix is to count deliveries somewhere a crash can't erase, with one counter per partition and offset. The consumer increments it before handling a message and, at 5 deliveries, sends the message to the DLQ unprocessed.
Which failure should the loader send to trip_payments_dlq?
The envelope
| Field | Why replay needs it |
|---|---|
| original_payload | Replay sends these exact bytes back through the loader; a re-serialized copy can hide the bug |
| source topic, partition, offset | Locates the message in the source and shows whether it was ever committed |
| error class and type | Groups 4,000 entries into 3 causes instead of 4,000 stack traces |
| consumer_version | Tells you whether a fix has shipped since, so a replay is worth trying |
Retention longer than the source
The envelope carries a copy of the payload because the source won't keep one. If trip_payments retains messages for 7 days, the original of offset 918223 is deleted on day 8, and from then on the DLQ holds the only copy of that payment. DLQ retention therefore has to cover the slowest realistic path from failure to replay.
| Step | Slow but realistic |
|---|---|
| Entry lands Friday 18:00, alert read Monday morning | 3 days |
| Root cause found, fix reviewed and deployed | 4 days |
| Replay run and row counts checked | 1 day |
| Total from failure to replay | 8 days |
- Alert on DLQ arrivals and on the age of the oldest entry
- Name an owner: the team that owns the consumer
- Replay through the same loader code, so validation runs again
- Leave the DLQ unwatched: a queue nobody reads is a delete with a delay
- Edit payloads in place; publish a corrected copy and keep the original
DLQ entries hold only an error string and a timestamp. The decimal-comma fix ships 12 days after the first entry; trip_payments keeps messages 7 days. What happens to the first day's failed payments?
4 limits around a retry loop
| Limit | What it bounds | Loader setting |
|---|---|---|
| Max attempts | Calls one message makes to the warehouse | 6 |
| Base delay and cap | The length of any single wait | 0.5 s, doubling, capped at 8 s |
| Elapsed deadline | Total time a message can hold a worker | 20 s |
| Jitter | How many workers retry at the same instant | Full: a random wait from 0 to the computed delay |
Walking the worst case
| Attempt | Starts | Fails | Wait before next |
|---|---|---|---|
| 1 | 0.0 s | 2.0 s | 0.5 s |
| 2 | 2.5 s | 4.5 s | 1 s |
| 3 | 5.5 s | 7.5 s | 2 s |
| 4 | 9.5 s | 11.5 s | 4 s |
| 5 | 15.5 s | 17.5 s | 8 s |
| 6 | Would start at 25.5 s | Would end at 27.5 s | Refused: past the 20 s deadline |
Only TransientError is caught. A permanent error, such as the 12,50 parse failure, passes straight through on the first attempt, which is how it reached the DLQ with attempts: 1.
A loader's budget is 10 max attempts, a 1 s base delay that doubles, a 4 s cap, a 5 s timeout per attempt, and no elapsed deadline. If every attempt hangs until its timeout, how long can a single message hold a worker?
Jitter sized to what an attempt costs
The minimum window is workers * seconds_per_attempt / concurrent_slots. Take a downstream that serves 20 requests at once. A currency lookup that holds a connection for 50 ms needs 200 * 0.05 / 20 = 0.5 s of spread. A bulk load that holds a warehouse slot for 30 s needs 200 * 30 / 20 = 300 s. The 0.5 s base delay that spreads the lookups well would start all 200 bulk loads within the same half-second, 10 times what the warehouse accepts.
After a network blip, 120 backfill workers fail at the same moment. Each retry runs a 10 s query against a database that allows 40 concurrent queries. What is the smallest jitter window that keeps around 40 queries running at once?
When retries become the outage
A retry budget bets that the next attempt might work. During a long outage that bet loses every time. At 18:10 the currency service that enrich_fx calls to convert EUR fares to USD stops answering for 10 minutes. The enricher uses the same budget as the loader, so each worker spends 5 attempts and 17.5 s on a message, leaves its offset uncommitted, and starts over on the same message. No payment gets through, and the service absorbs a steady flood of calls while it tries to restart.
A circuit breaker sits between a caller and a single dependency, watches the outcomes of recent calls, and stops calling once failures cross a threshold. While it's open, every call fails at once with CircuitOpen. That failure takes microseconds and sends nothing over the network, so the struggling service gets quiet in which to recover.
Closed, open, half-open
| State | What a call does | Next state | When |
|---|---|---|---|
| Closed | Goes through; its outcome joins a window of the last 20 calls | Open | At least 10 of the last 20 calls failed |
| Open | Fails at once with `CircuitOpen`, no network call | Half-open | The 30 s cooldown has passed |
| Half-open | Lets 3 trial calls through; the rest fail at once | Closed | All 3 trials succeed |
| Half-open | Same trial rule | Open | Any trial fails; the 30 s cooldown restarts |
Why each number is there
- A window of 20 callsTrips on evidenceWithout a minimum, 1 failure in the first 2 calls reads as 50%.
- A 50% failure rateTrips on outagesA few slow calls leave it closed; a dead service trips it in seconds.
- A 30 s cooldownGives recovery roomLong enough for a restart to finish, short enough to notice recovery within a minute.
- Trials of 3 callsProbes gentlyA single success could be luck; 3 in a row is evidence, and far short of a stampede.
A breaker opens at 14:00:00 with a 30 s cooldown and 3 trial calls. At 14:00:31 the first trial succeeds and the second times out. What happens next?
What the pipeline does with CircuitOpen
The code that catches CircuitOpen decides what happens next. The error says nothing about the message, so it is classified as transient and the message never goes to the DLQ. The enricher stops polling and leaves its committed offset where it is. After the cooldown, allow_request lets it take up to 3 messages from that offset as the half-open trials. Once all 3 succeed, the breaker closes; consumption picks up from that offset, so no payment is lost or skipped. A single failed trial pauses the enricher again.
This function runs inside call_with_budget, so every retry attempt counts toward the failure rate. CircuitOpen is deliberately a separate error from TransientError: the retry loop does not catch it, so an open breaker ends the loop on the spot instead of sleeping through 5 attempts that are certain to fail.
- Scope: a single call and its attempts
- Expects the next attempt to work
- Adds load: up to 6 calls per message
- Fits blips that last seconds
- Scope: every call to a dependency
- Expects failure until a trial succeeds
- Removes load: 0 calls while open
- Fits outages that last minutes
A teammate changes the handler so that CircuitOpen sends the message to trip_payments_dlq, since a fast failure is still a failure. What does the next 10-minute currency outage do?
A file of 10,000 rows with 37 bad ones
At 02:00 on Sunday the payouts job loads a file of driver payouts into the 2026-10-24 partition of fact_driver_payouts. The file has 10,000 rows, and 37 of them have an amount_raw that won't cast to a decimal. A batch has to decide what the partition holds as a whole, and there are 3 answers.
| Strategy | Partition this morning | The failures | Rerun after the fix |
|---|---|---|---|
| All-or-nothing | Empty; the swap never ran | Run blocked, owner paged | All 10,000 rows |
| Skip-and-quarantine | 9,963 rows | 37 rows in `quarantine_driver_payouts` | The 37 rows |
| Partial commit, 10 chunks of 1,000 | 8,000 rows from 8 chunks | Chunks 3 and 8 marked failed in a run ledger | 2,000 rows in 2 chunks |
All-or-nothing: a partial answer is a wrong answer
fact_driver_payouts feeds the bank transfer file. A partition with 9,963 rows looks healthy: it has a total, and 37 payouts are missing with nothing on any dashboard to say so. When a consumer acts on a partial output as if it were complete, the batch must land whole or not at all. The job loads into staging_driver_payouts first and swaps the partition in only after a check passes.
The task running this query fails when bad_rows is above 0, so the swap task downstream of it never runs and every earlier partition stays as it was. Here the rerun is cheap, a 4-minute load, so discarding 9,963 good rows costs almost nothing.
Skip-and-quarantine: a gap you can name
The load into fact_driver_payouts uses the opposite filter, IS NOT NULL. Both statements are preceded by a delete of the 2026-10-24 rows they wrote earlier, so a rerun replaces its output instead of appending to it. When the 37 rows are corrected, they are merged into the partition on payout_id, the business key, so a second fix attempt cannot double them.
Partial commit: a run too costly to repeat
| chunk_id | rows | status | finished_at |
|---|---|---|---|
| 1 | 1 to 1,000 | committed | 02:25 |
| 2 | 1,001 to 2,000 | committed | 02:50 |
| 3 | 2,001 to 3,000 | failed | 03:15 |
| 4 to 7 | 3,001 to 7,000 | committed | 04:55 |
| 8 | 7,001 to 8,000 | failed | 05:20 |
| 9 to 10 | 8,001 to 10,000 | committed | 06:10 |
Partial commit needs 2 things the other strategies do not: the ledger, and writes that replace by chunk_id, so a chunk that half-wrote before failing can run again without doubling rows. The resumed run reads the ledger and processes only chunks 3 and 8, taking 50 minutes instead of 250. The price is consistency: until both chunks land, the partition holds 8,000 of 10,000 rows, and consumers must either accept that or wait for the ledger to show all 10 chunks committed.
A chunked job writes each chunk with a plain INSERT, then marks it committed in the ledger. Chunk 6's insert succeeds, then the job crashes before the ledger update. What does the resumed run do?
The decision as a grid
| Consumers of the output | Rerun is cheap | Rerun is expensive |
|---|---|---|
| Act on totals: payments, ledgers, tax filings | All-or-nothing | Partial commit into staging; publish once every chunk commits |
| Tolerate a recorded gap: dashboards, exploration | Skip-and-quarantine | Partial commit with a quarantine for bad rows |
A nightly job builds daily_revenue_by_city, which finance uses to file city sales-tax remittances. The run takes 6 minutes. Last night 12 of 400,000 rows had a null city. Which strategy fits?
The third kind: ambiguous
| Failure | Example here | Recovery path |
|---|---|---|
| Transient, short | Warehouse returns 503 for 2 s | Retry inside the budget |
| Transient, sustained | Currency service down 18:10 to 18:20 | Breaker opens; consumer holds its offset and resumes |
| Permanent, single message | Amount `12,50` | DLQ with envelope; consumer commits past it |
| Permanent, rows in a batch | 37 payout rows fail the cast | Gate blocks the swap; owner fixes and reruns |
| Ambiguous | Write times out after it committed | Retry, made safe by MERGE on `payment_id` |
Payments stream from validate to enrich_fx, then into load_payments; payouts arrive nightly as a file and pass a gate before the swap. A retryCount of 5 means 6 attempts in all.
Rules that make the layers compose
The currency service starts returning 400 unsupported currency XAF for 0.2% of payments, and the classifier has no rule for it. What happens, and what's the fix?
Friday's 10 minutes, layer by layer
| Time | What happens | Layer that acts | Outcome |
|---|---|---|---|
| 18:00:04 | Offset 918223 carries amount `12,50` | validate | DLQ entry written, offset committed |
| 18:06:41 | Warehouse returns 503 for 2 s | load_payments retry | Succeeds on a later attempt |
| 18:10:02 | Currency service stops answering | enrich_fx breaker | Opens once 10 of its last 20 calls fail |
| 18:10:32 | First half-open trial times out | enrich_fx breaker | Reopens for another 30 s |
| 18:20:32 | All 3 trials succeed | enrich_fx breaker | Closes; consumer resumes from its held offset |
| 18:20:35 | Write for payment p-7120 times out after committing | load_payments retry | MERGE finds the row; no duplicate |
To speed up loading, a teammate replaces the MERGE on payment_id in load_payments with a plain INSERT. Which row of the Friday timeline now corrupts fact_payments?
> Uber's driver payments pipeline on a Friday evening: a fare event with a decimal comma, a 10-minute outage of the currency service, a warehouse write that times out after it committed, and a nightly payouts file with 37 unparseable amounts. By morning each failure sits in a named place, and no driver is paid twice or left unpaid.
payment_id turns the 18:20:35 timeout into a harmless retry, the same replace-instead-of-append rule the payouts job follows.CircuitOpen pauses consumption and never reaches the DLQ.Retry it, wait it out, set it aside or stop the run: a working model for sending each failure to the right place.
- Category
- Pipeline Architecture
- Difficulty
- intermediate
- Duration
- 32 minutes
- Challenges
- 12 hands-on challenges
Topics covered: Dead Letter Queue Basics, Retry Budgets: Max, Delay, Jitter, Circuit Breakers Stop the Hammer, Partial Failure in a Batch, Pipeline Handling All Three
Lesson Sections
- Dead Letter Queue Basics (concepts: Dead Letter Queues)
A bad message blocks a whole partition A dead letter queue (DLQ) is a second destination for messages the consumer has classified as unprocessable, each stored with the reason it failed. After writing the message to the DLQ, the consumer commits past it and moves on. The bad record is kept for a person to inspect, and the good records behind it load within seconds instead of waiting days for a fix. What goes in, and what stays out The left column holds failures that belong to the message: replay
- Retry Budgets: Max, Delay, Jitter (concepts: Retry & Failure Handling)
4 limits around a retry loop The beginner lesson gave retries a delay formula: wait longer after each failure. A retry budget is the set of limits around that formula, and each limit stops a different quantity from growing without bound. The payments loader's warehouse write uses the budget below, plus a timeout of 2 s on every attempt so a hung call cannot stretch a single attempt into minutes. Walking the worst case In the worst case every attempt hangs until its 2 s timeout. With jitter switc
- Circuit Breakers Stop the Hammer
When retries become the outage Closed, open, half-open Each cooldown ends with 3 trial calls, a load any restarting service can take. When the service returns at 18:20, all 3 calls in the next probe succeed and the breaker closes. Traffic resumes within 30 s of the recovery. Why each number is there What the pipeline does with CircuitOpen
- Partial Failure in a Batch (concepts: Dead Letter Queues)
A file of 10,000 rows with 37 bad ones All-or-nothing: a partial answer is a wrong answer Skip-and-quarantine: a gap you can name Suppose instead the file fed only a driver-earnings dashboard that the operations team reviews weekly. A gap of 0.37% is acceptable there, as long as it is recorded and later filled. The job writes failing rows to a quarantine table with a reason, then loads the rest. A threshold still guards against a broken file: when more than 1% of rows fail, the file itself is su
- Pipeline Handling All Three (concepts: Retry & Failure Handling)
The third kind: ambiguous The chunk 6 crash from the ledger question is a third kind of failure, an ambiguous one: the caller cannot tell whether the operation took effect. A warehouse write that times out after 2 s may have committed a millisecond before the connection dropped. Retrying it is safe only when the write is idempotent; otherwise the retry turns uncertainty into a duplicate payment. Rules that make the layers compose Friday's 10 minutes, layer by layer Every row of the Outcome colum