Failure Classification by Design
Concepts covered: Retry & Failure Handling
Splitting failures into transient and permanent answers a single question: retry or not. A production pipeline has to answer 4 questions for every failure. Should it retry? Should it stop calling the dependency? Where does the failed record go? Does someone get woken up? The answers to all 4 depend on 2 properties of the failure: how long it will last, and how much of the traffic it hits.
Every example in this lesson uses the same pipeline. It reads 2,000 order events per second from a durable log, calls a tax service that can serve 3,000 requests per second, and writes each priced order to curated.orders, keyed on order_id.
Duration and scope
Scope is the newer idea. A record-scoped failure hits a single record because of something inside it: a malformed amount, a missing customer ID. A systemic failure hits every record because of something outside them all: an overloaded service, an expired credential, a bug in freshly deployed code. The same error can land on either side. A timeout on 1 call in 10,000 is record-scoped noise, while timeouts on every call are systemic.
| Class | Duration | Scope | Example | Response |
|---|---|---|---|---|
| Transient record | Seconds | Single record | A lone 503 or connection reset | Retry in place with backoff and jitter |
| Transient systemic | Minutes to hours | All records | Tax service overloaded, 429 on every call | Open the breaker, leave records in the source log |
| Permanent record | Forever | Single record | `tip_amount` holds the string 'abc' | Write it to the DLQ and keep going |
| Permanent systemic | Until a human acts | All records | Expired credential returning 401; a bug in the transform | Halt the pipeline and page the owner |
The bottom row is where pipelines most often go wrong. Record by record, a permanent systemic failure looks like a permanent record failure: the call returns 401 and the order cannot be priced. A pipeline that checks only the error type sends every order to the DLQ, and an hour later the DLQ holds 7.2 million good orders. All of them need the same fix, a new credential, and none of them needed a decision of its own.
The classifier is code, and it reads the failure rate
Classification belongs in a single function that every retry, breaker and DLQ decision calls, so the rules live in one place and can be tested. It takes 2 inputs: the error, and the share of recent records that failed with the same kind of error. For timeouts and overload responses, that share tracks the tax service. For parse errors, it tracks the payloads. That share is what promotes a record-scoped verdict to a systemic one.
from dataclasses import dataclass
TRANSIENT_STATUS = {429, 502, 503, 504}
AUTH_STATUS = {401, 403}
SYSTEMIC_RATE = 0.5
@dataclass
class Verdict:
failure_class: str
action: str
def classify(error, same_kind_failure_rate):
status = getattr(error, "status", None)
systemic = same_kind_failure_rate >= SYSTEMIC_RATE
if isinstance(error, DuplicateKeyError):
return Verdict("already_applied", "ack")
if status in AUTH_STATUS:
return Verdict("permanent_systemic", "halt_and_page")
if status in TRANSIENT_STATUS or isinstance(error, (TimeoutError, ConnectionError)):
if systemic:
return Verdict("transient_systemic", "open_breaker")
return Verdict("transient_record", "retry_with_backoff")
if isinstance(error, (ValueError, KeyError)):
if systemic:
return Verdict("permanent_systemic", "halt_and_page")
return Verdict("permanent_record", "send_to_dlq")
return Verdict("unknown", "retry_then_dlq")
A duplicate-key error on an insert-if-absent write keyed on order_id and event_seq together means an earlier attempt already landed this exact event, so the record counts as a success and is acknowledged. A ValueError on half of all records points to a deployed bug or a producer format change, so the per-record verdict is overridden and the pipeline halts. An error the function has never seen gets a short retry and then the DLQ, and the count of unknown verdicts is tracked: when it climbs past a handful per hour, the classifier needs a new rule.
One table drives retry, breaker, queue and runbook
With classes defined, every mechanism downstream reads from the same table. That includes the runbook, the page an on-call engineer opens when an alert fires. It gets an entry per class instead of an entry per error message, which keeps it short enough to use at 3 a.m.
| Class | Retry | Breaker | Record goes to | Pages | Runbook first step |
|---|---|---|---|---|---|
| Transient record | Up to 4 attempts, exponential backoff, jitter | Counts toward failure rate | DLQ after attempt 4 | No | None |
| Transient systemic | Paused while breaker is open | Opens at 50% failures over 30 s | Stays in the source log | After 10 min open | Check the dependency's health and recent deploys |
| Permanent record | None | Not counted | DLQ with error signature | No | Group the DLQ by signature |
| Permanent systemic | None | All calls stopped | Stays in the source log | Immediately | Check the last deploy and credential expiry |
| Already applied | None | Not counted | Acknowledged | No | None |
A credential rotation at the tax service goes wrong, and from 14:00 every call from the order pipeline returns 401. Today the pipeline sends any record that fails 4 attempts to the DLQ. What should it do instead?
- Retry each record 4 times with backoff, then write it to the DLQ for later replay
- Leave each record in the source log, stop calling the tax service, and page the owner
- Open the breaker and probe every 30 seconds, sending records to the DLQ while it waits
- Write each record to the DLQ at once without retrying, because a 401 is permanent
About this lesson
This section is part of the Failure Handling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.