Pipeline Handling All Three
Concepts covered: Retry & Failure Handling
The third kind: ambiguous
The chunk 6 crash from the ledger question is a third kind of failure, an ambiguous one: the caller cannot tell whether the operation took effect. A warehouse write that times out after 2 s may have committed a millisecond before the connection dropped. Retrying it is safe only when the write is idempotent; otherwise the retry turns uncertainty into a duplicate payment.
| Failure | Example here | Recovery path |
|---|---|---|
| Transient, short | Warehouse returns 503 for 2 s | Retry inside the budget |
| Transient, sustained | Currency service down 18:10 to 18:20 | Breaker opens; consumer holds its offset and resumes |
| Permanent, single message | Amount `12,50` | DLQ with envelope; consumer commits past it |
| Permanent, rows in a batch | 37 payout rows fail the cast | Gate blocks the swap; owner fixes and reruns |
| Ambiguous | Write times out after it committed | Retry, made safe by MERGE on `payment_id` |
- trip_payments (queue, message queue)
- payouts_file (source, file drop)
- fx_rates (custom, currency service)
- validate (transform, consumer); on error: parse or rule failure: DLQ
- enrich_fx (transform, consumer); on error: breaker open: hold offset, pause; retries: 5; backoff: 0.5 s doubling, 8 s cap, full jitter
- load_payments (transform, consumer); retries: 5; idempotency: MERGE on payment_id
- payouts_job (transform, orchestrator); on error: any bad row: block the swap, page owner; idempotency: delete and reload the partition
- fact_payments (storage, warehouse); freshness: 15 min
- fact_driver_payouts (storage, warehouse)
- trip_payments_dlq (queue, message queue); alerting: arrivals and oldest-entry age
- trip_payments -> validate
- validate -> enrich_fx
- fx_rates -> enrich_fx
- enrich_fx -> load_payments
- load_payments -> fact_payments
- validate -> trip_payments_dlq
- enrich_fx -> trip_payments_dlq
- payouts_file -> payouts_job
- payouts_job -> fact_driver_payouts
- Sources: trip_payments, payouts_file, fx_rates
- Processing: validate, enrich_fx, load_payments, payouts_job
- Outputs: fact_payments, fact_driver_payouts
- Failure store: trip_payments_dlq
Rules that make the layers compose
- Classify first: every error maps to transient, permanent or ambiguous before any handler runs.
- The breaker is checked on every attempt inside the retry loop, so each attempt counts toward its failure rate.
- An open breaker stops the retry loop and pauses consumption; it never writes to the DLQ.
- Every write that can be retried is idempotent, because a timeout can hide a success.
- An error nobody has classified gets the retry budget, then goes to the DLQ marked unknown.
PERMANENT = (AmountParseError, RuleViolation, UnknownSchemaVersion)
TRANSIENT = (ServiceUnavailable, ConnectionReset, RateLimited)
AMBIGUOUS = (WriteTimeout,)
def recovery_path(error):
if isinstance(error, CircuitOpen):
return "pause"
if isinstance(error, PERMANENT):
return "dead_letter"
if isinstance(error, AMBIGUOUS):
return "retry_idempotent"
if isinstance(error, TRANSIENT):
return "retry"
return "retry_then_dead_letter_unknown"
The currency service starts returning 400 unsupported currency XAF for 0.2% of payments, and the classifier has no rule for it. What happens, and what's the fix?
- The breaker opens and pauses the pipeline; raise its failure threshold
- They exhaust the retry budget and reach the DLQ as unknown; add a permanent rule
- The payments vanish without a trace; add a log line to catch them
- The payments block their partition for good; add an elapsed deadline
Friday's 10 minutes, layer by layer
| Time | What happens | Layer that acts | Outcome |
|---|---|---|---|
| 18:00:04 | Offset 918223 carries amount `12,50` | validate | DLQ entry written, offset committed |
| 18:06:41 | Warehouse returns 503 for 2 s | load_payments retry | Succeeds on a later attempt |
| 18:10:02 | Currency service stops answering | enrich_fx breaker | Opens once 10 of its last 20 calls fail |
| 18:10:32 | First half-open trial times out | enrich_fx breaker | Reopens for another 30 s |
| 18:20:32 | All 3 trials succeed | enrich_fx breaker | Closes; consumer resumes from its held offset |
| 18:20:35 | Write for payment p-7120 times out after committing | load_payments retry | MERGE finds the row; no duplicate |
Every row of the Outcome column ends in a named state. Messages that are set aside have an owner; the rest are committed or retried, or wait while the consumer is paused. None ends with a payment dropped or doubled.
To speed up loading, a teammate replaces the MERGE on payment_id in load_payments with a plain INSERT. Which row of the Friday timeline now corrupts fact_payments?
- The 18:00:04 row, because the DLQ path writes the payment twice
- The 18:06:41 row, because the 503 retry writes the payment twice
- The 18:20:35 row, because the retried write adds a second copy
- The 18:20:32 row, because resumed messages are written twice
About this lesson
This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.