Pipeline Handling All Three

Concepts covered: Retry & Failure Handling

The third kind: ambiguous

The chunk 6 crash from the ledger question is a third kind of failure, an ambiguous one: the caller cannot tell whether the operation took effect. A warehouse write that times out after 2 s may have committed a millisecond before the connection dropped. Retrying it is safe only when the write is idempotent; otherwise the retry turns uncertainty into a duplicate payment.

FailureExample hereRecovery path
Transient, shortWarehouse returns 503 for 2 sRetry inside the budget
Transient, sustainedCurrency service down 18:10 to 18:20Breaker opens; consumer holds its offset and resumes
Permanent, single messageAmount `12,50`DLQ with envelope; consumer commits past it
Permanent, rows in a batch37 payout rows fail the castGate blocks the swap; owner fixes and reruns
AmbiguousWrite times out after it committedRetry, made safe by MERGE on `payment_id`
Payments stream from validate to enrich_fx, then into load_payments; payouts arrive nightly as a file and pass a gate before the swap. A retryCount of 5 means 6 attempts in all.
  • trip_payments (queue, message queue)
  • payouts_file (source, file drop)
  • fx_rates (custom, currency service)
  • validate (transform, consumer); on error: parse or rule failure: DLQ
  • enrich_fx (transform, consumer); on error: breaker open: hold offset, pause; retries: 5; backoff: 0.5 s doubling, 8 s cap, full jitter
  • load_payments (transform, consumer); retries: 5; idempotency: MERGE on payment_id
  • payouts_job (transform, orchestrator); on error: any bad row: block the swap, page owner; idempotency: delete and reload the partition
  • fact_payments (storage, warehouse); freshness: 15 min
  • fact_driver_payouts (storage, warehouse)
  • trip_payments_dlq (queue, message queue); alerting: arrivals and oldest-entry age
  • trip_payments -> validate
  • validate -> enrich_fx
  • fx_rates -> enrich_fx
  • enrich_fx -> load_payments
  • load_payments -> fact_payments
  • validate -> trip_payments_dlq
  • enrich_fx -> trip_payments_dlq
  • payouts_file -> payouts_job
  • payouts_job -> fact_driver_payouts
  • Sources: trip_payments, payouts_file, fx_rates
  • Processing: validate, enrich_fx, load_payments, payouts_job
  • Outputs: fact_payments, fact_driver_payouts
  • Failure store: trip_payments_dlq

Rules that make the layers compose

PERMANENT = (AmountParseError, RuleViolation, UnknownSchemaVersion)
TRANSIENT = (ServiceUnavailable, ConnectionReset, RateLimited)
AMBIGUOUS = (WriteTimeout,)


def recovery_path(error):
    if isinstance(error, CircuitOpen):
        return "pause"
    if isinstance(error, PERMANENT):
        return "dead_letter"
    if isinstance(error, AMBIGUOUS):
        return "retry_idempotent"
    if isinstance(error, TRANSIENT):
        return "retry"
    return "retry_then_dead_letter_unknown"

The currency service starts returning 400 unsupported currency XAF for 0.2% of payments, and the classifier has no rule for it. What happens, and what's the fix?

Friday's 10 minutes, layer by layer

TimeWhat happensLayer that actsOutcome
18:00:04Offset 918223 carries amount `12,50`validateDLQ entry written, offset committed
18:06:41Warehouse returns 503 for 2 sload_payments retrySucceeds on a later attempt
18:10:02Currency service stops answeringenrich_fx breakerOpens once 10 of its last 20 calls fail
18:10:32First half-open trial times outenrich_fx breakerReopens for another 30 s
18:20:32All 3 trials succeedenrich_fx breakerCloses; consumer resumes from its held offset
18:20:35Write for payment p-7120 times out after committingload_payments retryMERGE finds the row; no duplicate

Every row of the Outcome column ends in a named state. Messages that are set aside have an owner; the rest are committed or retried, or wait while the consumer is paused. None ends with a payment dropped or doubled.

To speed up loading, a teammate replaces the MERGE on payment_id in load_payments with a plain INSERT. Which row of the Friday timeline now corrupts fact_payments?

About this lesson

This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.