Failure Modes and Error Handling: Intermediate
A retail data pipeline at a fintech consumed payment events from Kafka and wrote them to Snowflake. One Wednesday a downstream verification API began returning 503s for seven percent of requests. The pipeline had a single retry with no backoff. Within four minutes the API was returning 503s for one hundred percent of requests because the retry storm had taken it down. The outage lasted two hours and fourteen minutes. The post-incident review identified three missing patterns. There was no dead letter queue, so messages that could not be processed kept being processed. There was no circuit breaker, so the pipeline kept hammering the API even after every recent request had failed. There was no per-row failure handling, so a single bad message failed an entire batch of ten thousand. This lesson is about the patterns that turn the basic retry into a complete failure handling strategy. The patterns are the difference between a pipeline that survives a partner outage and one that becomes the partner outage.
Dead Letter Queue Basics
Design a dead letter queue with the right envelope, retention, and operational discipline so failed messages are recoverable rather than lost.
A retry exhausts its budget and the message still has not been processed. The pipeline now faces a choice. It can drop the message, which loses data silently. It can crash and stop processing, which blocks every other message behind it. Or it can move the message somewhere else, somewhere a human can look at it later, while the pipeline continues processing the rest. The third option is the dead letter queue. The dead letter queue is the conventional name for the side channel that holds messages a pipeline could not handle. The mechanism is simple; the discipline of using it correctly is what separates production-ready pipelines from research code.
What a DLQ Actually Is
A dead letter queue is a separate destination, usually another Kafka topic, an SQS queue, or a database table, where the pipeline writes messages it could not process after exhausting its retry budget. The structure is the same as the main queue but the consumer is different. Instead of the regular pipeline workers, the consumer is either a human, a tool that surfaces messages for inspection, or a separate process that knows how to attempt recovery. The DLQ is durable storage. Messages sitting in it are not lost, but they are also not yet processed.
| DLQ Implementation | Typical Use | Trade-off |
|---|---|---|
| A second Kafka topic | Streaming pipelines, message-based architectures | Same tooling, same retention rules; consumer must be built separately |
| An SQS or Kinesis queue | AWS-native pipelines, Lambda triggers | Fully managed; AWS controls many of the operational details |
| A relational table | Batch ETL pipelines, dbt-flavored failures | Easy to query, easy to inspect; harder to drain at scale |
| An object store path | File-based pipelines, large failed payloads | Cheapest storage; hardest to operate against without tooling |
What the Pipeline Writes to a DLQ
A useful DLQ entry contains more than the failed message. It contains the original payload, the timestamp of the failure, the exception type and message, the number of attempts that were made, the worker identity, and a correlation ID that ties the failed message back to its origin. Without that context, draining the DLQ becomes a forensic exercise. With that context, an engineer can inspect a failed message in seconds, decide whether the upstream needs a fix or the message itself was malformed, and act accordingly.
When a Message Should Go to the DLQ
•Goes to DLQ
- Retry budget exhausted on transient errors
- Permanent error classified at the message level
- Schema mismatch that cannot be parsed
- Validation failure on the row payload
✓Stays in Main Queue
- Transient error within the retry budget
- Service-level outage where every message is failing
- Errors caused by a downstream that has not yet recovered
- Errors clearly attributable to a deploy that is being rolled back
The right column is important. A DLQ is for failures specific to a message. A DLQ is not where messages go when the entire downstream is unavailable. If every message is failing because Snowflake is in a maintenance window, routing them all to a DLQ produces a million-row DLQ that has to be drained as soon as Snowflake comes back. The correct response to a service-wide outage is to stop consuming and let the queue accumulate, not to push the queue into the DLQ. Distinguishing message-specific from service-wide failures is the same classification problem from the beginner tier, applied at a slightly higher level.
DLQ design checklist:
- ▸The DLQ has the same durability and retention as the main queue
- ▸The envelope captures original payload, exception, attempts, and timestamp
- ▸There is a tool that lets a human read DLQ entries without writing custom code
- ▸There is a clear runbook for replaying a fixed message back into the main pipeline
The DLQ Is Not a Drop
A DLQ that nobody ever drains is functionally identical to dropping the message. The only difference is the appearance of doing the right thing while in fact doing the same wrong thing. A real DLQ has someone responsible for monitoring its size, an alert when it grows beyond an expected baseline, and a defined process for handling each entry. Without those operational properties, the DLQ becomes a graveyard. Healthy data engineering organizations treat DLQ depth as a first-class metric and pay for the on-call attention it requires.
The envelope shape is the durable contract between the producer side of the pipeline and any future replay tool. Adding fields later is cheap; removing fields is the kind of breaking change a contract should prevent.
A DLQ is durable storage for messages that could not be processed after retries.
The envelope captures enough context for forensics; the original payload alone is not enough.
DLQ depth is a first-class operational metric; an unread DLQ is identical to a drop.
DLQ retention is a design choice that often gets overlooked. A DLQ with one-day retention turns into a drop the day before a long weekend. A DLQ with infinite retention turns into a storage cost that nobody is responsible for. The right answer is workload-dependent: a streaming pipeline whose DLQ entries can be replayed within a day might use a three-day retention. A batch pipeline whose DLQ entries require human inspection across multiple business days might use a two-week retention. The retention should match the operational rhythm of the team that drains the DLQ, not the default of whatever queueing system was selected.
TIP
Define the DLQ alert threshold the same week the DLQ ships. The window between shipping the DLQ and shipping its alerts is the window where messages get silently lost.
pipeline task
task
transient? retry w/ backoff
retry
warehouse
success
permanent -> DLQ
dlq
Classify the failure first: transient errors (timeout, lock) get retried with exponential backoff; permanent errors (bad schema) go straight to a dead-letter queue. Retrying a permanent error just wastes time.
Retry Budgets: Max, Delay, Jitter
Configure a retry budget with bounded attempts, bounded delay, bounded elapsed time, and jitter sized to the cost of the operation.
A retry budget is the explicit set of constraints that govern how a pipeline retries. The beginner tier defined the three numbers: maximum attempts, wait between attempts, and which errors retry. Production pipelines elaborate on those numbers with two more: a maximum cumulative delay across all attempts, and the jitter strategy used to desynchronize retry waves. A complete budget answers the question 'what is the worst case behavior of this retry policy' before the policy ever runs. Without that answer, retry behavior under stress is whatever the runtime decides.
The Five Numbers of a Retry Budget
| Parameter | What It Bounds | Common Default |
|---|---|---|
| max_attempts | Total tries including the first | 5 for low-cost calls, 3 for expensive calls |
| base_delay | First retry wait | 1 second |
| max_delay | Cap on any single retry's wait | 60 seconds for synchronous, 5 minutes for batch |
| max_total_elapsed | Hard ceiling on total time spent retrying | 10 minutes; if exceeded, give up regardless of attempt count |
| jitter_strategy | Method for randomizing waits across clients | Full jitter (uniform between 0 and the computed cap) |
Why max_total_elapsed Is Not Redundant
A budget with five attempts and a sixty-second cap allows a worst case of about 1 + 2 + 4 + 8 + 60 = 75 seconds of retries. That sounds bounded, but the budget interacts with jitter and with the underlying operation's own latency. A retry that times out after 30 seconds, retries with an additional 60-second jittered wait, times out again, and so on, ends up consuming far more wall clock time than the naive sum suggests. A separate max_total_elapsed clamp catches this case. After ten minutes, regardless of attempt count, regardless of computed wait, the retry gives up and the failure escalates.
Jitter Strategies Compared
| Strategy | Wait Formula | When to Use |
|---|---|---|
| No jitter | wait = min(cap, base * 2^attempt) | Single-client systems where synchronization is impossible |
| Full jitter | wait = uniform(0, min(cap, base * 2^attempt)) | Default; spreads retries across the entire window |
| Equal jitter | wait = half + uniform(0, half) where half = cap/2 | When a minimum wait is required to give downstream time to recover |
| Decorrelated jitter | wait = uniform(base, prev_wait * 3) capped at max | When the sequence of waits should be less predictable |
Full jitter is the AWS Architecture Blog recommendation and the default in most modern retry libraries. Decorrelated jitter is more aggressive about desynchronization at the cost of a wider variance in observed retry behavior. For most pipeline work the two perform indistinguishably; the choice rarely matters. What matters is that some jitter exists, because the alternative is a synchronized herd.
The Budget Has To Fit the Operation
✓Cheap Operation Budget
- max_attempts = 5
- base_delay = 1 second
- max_delay = 60 seconds
- max_total_elapsed = 5 minutes
•Expensive Operation Budget
- max_attempts = 3
- base_delay = 30 seconds
- max_delay = 5 minutes
- max_total_elapsed = 30 minutes
A REST call that costs nothing tolerates many quick retries. A Spark job that costs forty dollars to start does not. The retry budget for a long-running batch should have fewer attempts and longer waits, because the cost of each attempt is high and the optimization is in giving the underlying problem time to clear rather than in trying many times in close succession.
max_attemptsmax_delaymax_total_elapsed
max_attempts
Bounded retry count
5 for cheap calls, 3 for expensive ones. Higher than 8 is a sign the failure is not transient.
max_delay
Per-attempt cap
Prevents the exponential from running away. Sized to the on-call tolerance for stalled work.
max_total_elapsed
Cumulative ceiling
The clamp that catches the case where attempt count plus operation latency adds up to hours.
A retry budget that does not log every attempt makes the retry behavior invisible to operations. The log entry should include the operation name, the attempt number, the outcome, the elapsed time, and the exception type if the attempt failed. Aggregating these logs yields a per-operation view: which operations retry most often, which retry budgets are most often exhausted, which downstream services are degrading. Without this aggregation, the retry policy is a black box; with it, the retry policy becomes a continuous diagnostic signal.
Retry budgets interact with timeouts in non-obvious ways. A retry that fires after a one-second wait can still spend thirty seconds blocking on the underlying call before the timeout triggers. The total time spent in a single retry attempt is the sum of the wait and the timeout, not the wait alone. Pipelines that ignore this interaction end up with budgets that look bounded on paper but consume far more wall time than expected. The fix is to set the per-operation timeout deliberately, treat it as part of the budget, and verify the worst-case sum against the operational tolerance.
✓Do
- Define every budget parameter explicitly in code; never rely on library defaults
- Add jitter unconditionally; the cost is one line and the upside is preventing retry storms
- Log every attempt with attempt number, outcome, and elapsed time
✗Don't
- Set max_attempts higher than 8; if the work needs more, the pipeline needs more than retries
- Skip max_total_elapsed; without it, edge cases produce retries that run for hours
- Use the same budget for cheap REST calls and expensive Spark jobs
TIP
Treat the retry budget as a config object, not as scattered constants. A single named RetryBudget object that travels with the operation is the cleanest pattern in production code.
Circuit Breakers Stop the Hammer
Apply a circuit breaker to fail fast against a sustained downstream outage and design the closed/open/half-open state transitions.
Retries protect against momentary failures of a single request. A circuit breaker protects against sustained failures of an entire downstream service. The motivating problem is the case where every request is failing. A retry budget keeps issuing requests, each one more painful for the downstream than the last. The downstream has been overloaded for fifteen minutes; sending more requests is not helpful. The circuit breaker pattern, popularized by Michael Nygard's book Release It, says: if the downstream has been failing consistently for some window, stop calling it for a while. The pattern fits in a few lines of state but it is the difference between a partner outage that lasts twenty minutes and one that lasts six hours.
The Three States
| State | What the Pipeline Does | Transition Out |
|---|---|---|
| Closed | Calls the downstream normally; counts failures | If failures exceed the threshold, transition to open |
| Open | Refuses to call the downstream; fails fast | After a cool-down period, transition to half-open |
| Half-open | Allows a small number of trial requests | If trials succeed, transition to closed; if any fail, return to open |
Why Failing Fast Is the Win
An open breaker fails immediately without making the downstream call. The pipeline returns an error to the caller in milliseconds instead of after a thirty-second timeout. The downstream is no longer pressured by a flood of timing-out requests. Engineers monitoring the system see a clear signal: the breaker is open, the downstream is unhealthy, and recovery is in progress. Without a circuit breaker, the same outage manifests as a long tail of slow requests, all timing out in unpredictable patterns, with no clean signal that the underlying cause is the downstream rather than the pipeline.
How the Breaker Interacts With Retries
A retry policy and a circuit breaker work together rather than against each other. The retry handles a single request that might be transiently failing. The breaker handles the case where many requests have been failing recently. The interaction matters: the retry sits inside the breaker. The breaker decides whether to attempt the call at all, and if it does, the retry decides whether to try again. If the retry exhausts its budget, that counts as a failure for the breaker. Once the breaker is open, no requests are issued, no retries are spent, and the pipeline burns no more compute against a downstream that cannot serve.
•Without a Circuit Breaker
- Every request waits the full timeout before failing
- The pipeline keeps issuing requests during a sustained outage
- On-call sees thousands of slow failures with unclear cause
- The downstream has no chance to recover under continuous load
✓With a Circuit Breaker
- Requests fail in milliseconds once the breaker is open
- The pipeline stops issuing requests during a sustained outage
- On-call sees a clear breaker-open signal pointing at the downstream
- The downstream gets a quiet window to recover before traffic resumes
Sliding Windows in Real Implementations
The minimal breaker above counts consecutive failures. Production breakers count failures over a sliding window: the percentage of failed requests in the last sixty seconds, or the last fifty requests. The sliding-window form survives the case where a downstream is failing intermittently rather than continuously, which is the more common shape. Hystrix, the Netflix circuit breaker library, popularized this design. Modern equivalents include resilience4j and Polly. The library choice rarely matters; the design constraint is that the breaker counts over a window, not over consecutive events.
Tuning a circuit breaker:
- ▸failure_threshold: about 50% failure rate over the window is a common starting point
- ▸window: 30 to 60 seconds for synchronous calls; longer for batch
- ▸cooldown: long enough that the downstream has plausibly recovered, short enough that traffic resumes promptly
- ▸half-open trials: usually 1 to 3 to test before fully closing
An open breaker fails fast in milliseconds; a closed breaker passes calls through normally.
Half-open is the trial state that lets a small number of requests probe whether the downstream recovered.
Retry budgets sit inside breakers; a budget exhaustion counts as a failure toward the breaker's threshold.
Circuit breakers are sometimes confused with rate limiters. The two are different. A rate limiter caps the rate at which requests are sent regardless of whether the downstream is healthy. A circuit breaker stops sending requests entirely when the downstream is unhealthy. The two patterns compose: a healthy pipeline often has both, with the rate limiter sized to the downstream's normal capacity and the breaker sized to detect sustained degradation.
The half-open state is the part of the breaker design that engineers most often get wrong. Some implementations send all queued requests through immediately when the cooldown expires; the result is a smaller thundering herd against a downstream that may not have fully recovered. The correct half-open behavior is to send a small number of trial requests, observe the result, and only fully close the breaker after the trials succeed. A common rule of thumb is to allow one request per half-open transition, then close on success or re-open on any failure.
TIP
If a service has a circuit breaker around its calls, the cooldown should never be shorter than the alert delay for the downstream. A breaker that opens and closes faster than the alert can fire hides the outage.
Partial Failure in a Batch
Choose between all-or-nothing, skip-and-quarantine, and partial commit based on the consistency requirements of the output and the cost of reprocessing.
A batch job processes ten thousand rows. One row fails. The question is what happens to the other 9,999. The two extreme answers are both common and both wrong. Failing the entire batch loses progress on every good row. Silently dropping the bad row hides a problem that might be a symptom of a larger issue. The right answer is somewhere in the middle, and choosing the right point on the spectrum is one of the most consequential decisions a pipeline designer makes about a given workload.
Three Strategies
| Strategy | Behavior on Bad Row | Cost of Recovery |
|---|---|---|
| All-or-nothing | The whole batch fails; no rows are written | Reprocess the entire batch after the upstream fixes the bad row |
| Skip and quarantine | Bad row goes to a quarantine table; good rows are written | Inspect the quarantine table; reprocess just the bad rows |
| Partial commit | Good rows committed up to the failure; bad row aborts the rest | Resume from the failure point; depends on idempotency |
When All-or-Nothing Is the Right Answer
A financial reconciliation batch that produces a daily ledger should be all-or-nothing. The ledger needs to balance. A subset of the rows would produce a ledger that does not balance, which is worse than no ledger at all. The all-or-nothing strategy demands that the pipeline be idempotent, the property covered in Lesson 5, so that reprocessing the entire batch is safe. Without idempotency, all-or-nothing is dangerous because the partial state from the failed run can persist into the next run.
When Skip and Quarantine Is the Right Answer
A user analytics pipeline that aggregates page views across millions of events should skip and quarantine. One malformed event out of millions does not change any aggregate meaningfully. Failing the whole batch on one bad event means the rest of the analytics pipeline waits hours while a single row is investigated. The right answer is to keep the malformed event aside, log it, alert if the quarantine grows, and let the rest of the pipeline flow. The threshold for failing the whole batch is a percentage rule rather than an any-row rule: above a threshold of bad rows, fail; below it, quarantine.
When Partial Commit Is the Right Answer
A long-running ingestion that processes files sequentially benefits from partial commit. If the ten-thousandth file fails, the work on the first 9,999 should not be discarded. The pipeline records its progress, fails at the bad file, and on the next run resumes from where it left off. The pattern is sometimes called checkpoint-based recovery. It depends on the operation being checkpointable: the pipeline must be able to write down a structured record stating that the run successfully processed up through file 9,999 in a place that survives the failure. Without a checkpoint, partial commit is indistinguishable from all-or-nothing on the next run.
•Failing the Whole Batch
- Strict atomicity: either all rows succeed or none do
- Required when the output must be internally consistent
- Higher cost per failure (must reprocess everything)
- Cleanest semantic; easiest to reason about correctness
✓Failing One Row
- Lenient: bad rows quarantined, good rows written
- Required when most of the data is independent and one bad row should not block 9,999 good ones
- Lower cost per failure (only the bad rows need attention)
- More complex semantics; requires explicit threshold and quarantine plumbing
The Threshold Is a Real Decision
A skip-and-quarantine policy with no threshold is a silent failure waiting to happen. If 30% of rows in a batch are quarantined, something is wrong upstream and the pipeline should not be writing the remaining 70% as if everything is fine. The threshold names the line between 'this is normal noise' and 'this batch is broken.' Common thresholds are 1% to 5% depending on the workload. The threshold should be set at the time the policy ships, not discovered the first time a quarantine spike happens.
Choosing among the three:
- ▸All-or-nothing when the output must be internally consistent (ledgers, balanced books)
- ▸Skip and quarantine when most rows are independent and one bad row should not block the rest
- ▸Partial commit when the batch is long and the units of work are checkpointable
- ▸All three depend on idempotency to make recovery safe
Idempotency is the prerequisite, not the consequence. A non-idempotent pipeline that adopts skip-and-quarantine ends up with double-written rows the first time a batch is partially reprocessed. Lesson 5 covers the write patterns that make recovery safe.
Failing the whole batch on one row is sometimes correct, but only when the output must balance.
Skip and quarantine works for most analytics workloads; the threshold names the line between noise and a real problem.
Partial commit needs an idempotent checkpoint; otherwise the next run reprocesses everything anyway.
The threshold value chosen for skip-and-quarantine should be calibrated against historical data. A workload that historically produces 0.3% bad rows can safely set a 1% threshold. A workload that historically produces 0.05% bad rows could set a tighter 0.2% threshold and catch regressions faster. Setting the threshold without looking at history produces either constant false alarms (threshold too tight) or missed regressions (threshold too loose). The calibration takes one query against historical pipeline logs and is the highest-leverage tuning operation in the partial-failure design space.
TIP
Document the partial-failure strategy in the same place as the SLA. A team that does not know whether the pipeline is all-or-nothing or skip-and-quarantine cannot reason about its own data.
Pipeline Handling All Three
Compose retries, circuit breakers, DLQs, and quarantines into a single pipeline that handles transient, permanent, and ambiguous failures with named recovery paths.
Each pattern in isolation is straightforward. The hard part is composing them into a single pipeline that handles transient errors with backoff, permanent errors with a DLQ, and ambiguous errors with a bounded retry that escalates correctly. The example below is a streaming pipeline that consumes order events from Kafka, calls a downstream tax-calculation API, and writes the enriched events to Snowflake. It handles all three failure categories. Reading through the design end to end shows how the patterns reinforce each other.
The Architecture
The Failure Path Per Error Class
| Error | Path | Outcome |
|---|---|---|
| Tax API returns 503 once | Retry with backoff and jitter; second attempt succeeds | Event flows to Snowflake; no DLQ entry |
| Tax API returns 503 for sustained period | Retries exhaust; circuit breaker opens; events queue up | Pipeline pauses; on-call paged on breaker-open metric |
| Order event missing required field | Validation fails; classified as permanent | Event routed to orders_dlq with full envelope |
| Tax API returns 401 (bad token) | Classified as permanent; retry refused | Event routed to orders_dlq; alert fires for credential rotation |
| Tax API returns generic 500 | Classified as ambiguous; bounded retry of 3 attempts | If all attempts fail, event routes to orders_dlq |
The Code, Composed
Three lines of routing logic, three classes of failure, three destinations: the sink for success, the DLQ for permanent and budget-exhausted, and the implicit pause-and-retry for ambiguous transient errors that resolve within the budget. The breaker prevents the retry budget from being burned during a sustained outage. The DLQ catches everything the retry could not. The sink only sees enriched events that passed every check. No event is silently dropped; every failure is recoverable from the DLQ once the upstream cause is fixed.
Operational Properties of the Combined Design
Self-healing on transient errorsFail-fast on sustained outagesRecoverable on permanent errorsBounded blast radius
Self-healing on transient errors
Retries with backoff and jitter
503s, timeouts, and connection resets clear within the retry budget; on-call sleeps through the night.
Fail-fast on sustained outages
Circuit breaker around the tax API
When the API has been failing for thirty seconds, the breaker opens and stops sending requests. The downstream gets a recovery window.
Recoverable on permanent errors
DLQ for validation, auth, and exhaustion
Every event that fails for any non-transient reason lands in the DLQ with full context. Replay is straightforward once the cause is fixed.
Bounded blast radius
Retry budget plus jitter
No single event can consume more than a defined budget of compute. No retry wave from many workers can synchronize against the API.
The DLQ Drainage Tool
The replayer is the operational tool that turns the DLQ from a graveyard into a recovery surface. An engineer fixes the upstream credential, runs the replayer against the auth-failure entries, and the events flow through. A producer team patches a bug that was emitting malformed events, runs the replayer against the validation-failure entries, and the events flow through. The DLQ becomes a temporary holding area, not a permanent destination.
What this design promises:
- ▸No single event is silently dropped; everything is either delivered or recoverable
- ▸No transient downstream failure causes data loss
- ▸No sustained downstream failure burns retry budget endlessly
- ▸Every category of failure has a named path and a named recovery procedure
The handler above is intentionally short because the failure handling logic should not dominate the business logic of the worker. A handler that is more than a few dozen lines is a sign that the patterns have not been factored into reusable helpers. The shared library should encapsulate the breaker, the retry budget, the DLQ writer, and the classification logic; the worker only composes them. This separation matters because the worker code changes often as features are added, while the failure handling rarely changes once it is right. Mixing the two layers means every feature change risks regressing the failure handling.
The combined design above maps directly to canvas elements: the worker pool is a transform node, the Tax API call has a retry edge with a circuit breaker label, the DLQ topic is a queue node serving as the error sink, and the orders_dlq path is the error_path edge. Drawing the system on the canvas with these labels turns the architecture into operable documentation. Anyone who arrives on-call mid-incident can read the canvas and identify which path a given failed event followed without reading the worker source code. The same labeling discipline turns every architecture review meeting into a faster one because the picture and the runbook are already aligned.
The composition above also illuminates a subtle dependency on idempotency from Lesson 5. The replayer relies on the downstream sink being safe to write the same enriched event twice, because some replays will succeed at the source and fail at the sink, leading to apparent duplicate writes. Without idempotent sinks (partition overwrite, MERGE on a business key, delete-then-insert in a transaction), the replay produces duplicates. The patterns of this lesson and Lesson 5 are not orthogonal; they reinforce each other and a pipeline that has one without the other has a brittleness that will surface under any sustained failure.
✓Do
- Compose the patterns; never use them in isolation in a production pipeline
- Build the replayer the same week the DLQ ships; an unread DLQ is identical to a drop
- Surface breaker-open events as a first-class metric so on-call sees outages within seconds
✗Don't
- Skip the DLQ because 'this pipeline never has permanent failures'; it will, and the day it does the replayer will not exist
- Set the retry budget without reading the circuit breaker's window; the two interact
- Ship a DLQ without an alert on its depth; silence is not the same as success
❯❯❯PUTTING IT ALL TOGETHER
> A streaming pipeline at a logistics company consumes shipment events from Kafka and calls a third-party customs API to enrich them before writing to Snowflake. The pipeline has been crashing once or twice a day for a month. The current code has a retry with no jitter, no DLQ, no circuit breaker, and fails the whole batch on a single bad row. The team is asked to redesign the failure handling without rewriting the rest of the pipeline.
Step one: classify customs API errors. 503s and timeouts are transient; 401s and 422s are permanent; 500s are ambiguous and get a small bounded retry. The classification echoes the four-role separation from Lesson 1: a transform stage that owns its failure semantics.
Step two: replace the no-jitter retry with a full retry budget: five attempts, exponential backoff with full jitter, max delay of sixty seconds, max total elapsed of ten minutes. This sits inside the worker; it owns the per-request transient case.
Step three: add a circuit breaker around the customs API. Open after fifty percent failure rate over a thirty-second window. The breaker stops the retry budget from being burned during a sustained outage and gives the downstream a quiet window to recover.
Step four: add a DLQ Kafka topic. Validation failures, permanent classification errors, and budget-exhausted retries all route there with a full envelope. Build the replayer at the same time. The DLQ depends on idempotency from Lesson 5: replays must not produce duplicate enriched events.
Step five: change the batch failure mode from all-or-nothing to skip-and-quarantine with a one-percent threshold. A single malformed shipment no longer fails ten thousand good ones. Above the threshold, the batch fails and on-call investigates upstream.
KEY TAKEAWAYS
A DLQ is durable storage for failed messages: with a full envelope and a replayer, it turns failures into recoverable events instead of silent drops.
Retry budgets bound the worst case: max attempts, base delay, max delay, max total elapsed, jitter strategy. Defaults are fine; the discipline is making them explicit.
Circuit breakers protect against sustained outages: the closed/open/half-open state machine fails fast and gives downstreams a recovery window.
Partial failure strategy is a design decision, not a default: all-or-nothing for ledgers, skip-and-quarantine for analytics, partial commit for long batches with checkpoints.
Compose the patterns: retries, breakers, DLQs, and quarantines reinforce each other. A pipeline using only one of them has the failure mode the other three would have caught.
Retries are not enough; failed messages need a home and downstream services need protection
- Category
- Pipeline Architecture
- Difficulty
- intermediate
- Duration
- 32 minutes
- Challenges
- 0 hands-on challenges
Topics covered: Dead Letter Queue Basics, Retry Budgets: Max, Delay, Jitter, Circuit Breakers Stop the Hammer, Partial Failure in a Batch, Pipeline Handling All Three
Lesson Sections
- Dead Letter Queue Basics (concepts: paDeadLetterQueue)
A retry exhausts its budget and the message still has not been processed. The pipeline now faces a choice. It can drop the message, which loses data silently. It can crash and stop processing, which blocks every other message behind it. Or it can move the message somewhere else, somewhere a human can look at it later, while the pipeline continues processing the rest. The third option is the dead letter queue. The dead letter queue is the conventional name for the side channel that holds messages
- Retry Budgets: Max, Delay, Jitter (concepts: paRetryHandling)
A retry budget is the explicit set of constraints that govern how a pipeline retries. The beginner tier defined the three numbers: maximum attempts, wait between attempts, and which errors retry. Production pipelines elaborate on those numbers with two more: a maximum cumulative delay across all attempts, and the jitter strategy used to desynchronize retry waves. A complete budget answers the question 'what is the worst case behavior of this retry policy' before the policy ever runs. Without tha
- Circuit Breakers Stop the Hammer (concepts: paRetryHandling)
Retries protect against momentary failures of a single request. A circuit breaker protects against sustained failures of an entire downstream service. The motivating problem is the case where every request is failing. A retry budget keeps issuing requests, each one more painful for the downstream than the last. The downstream has been overloaded for fifteen minutes; sending more requests is not helpful. The circuit breaker pattern, popularized by Michael Nygard's book Release It, says: if the do
- Partial Failure in a Batch (concepts: paDeadLetterQueue)
A batch job processes ten thousand rows. One row fails. The question is what happens to the other 9,999. The two extreme answers are both common and both wrong. Failing the entire batch loses progress on every good row. Silently dropping the bad row hides a problem that might be a symptom of a larger issue. The right answer is somewhere in the middle, and choosing the right point on the spectrum is one of the most consequential decisions a pipeline designer makes about a given workload. Three St
- Pipeline Handling All Three (concepts: paRetryHandling)
Each pattern in isolation is straightforward. The hard part is composing them into a single pipeline that handles transient errors with backoff, permanent errors with a DLQ, and ambiguous errors with a bounded retry that escalates correctly. The example below is a streaming pipeline that consumes order events from Kafka, calls a downstream tax-calculation API, and writes the enriched events to Snowflake. It handles all three failure categories. Reading through the design end to end shows how the