Failure Classification by Design

Concepts covered: Retry & Failure Handling

Splitting failures into transient and permanent answers a single question: retry or not. A production pipeline has to answer 4 questions for every failure. Should it retry? Should it stop calling the dependency? Where does the failed record go? Does someone get woken up? The answers to all 4 depend on 2 properties of the failure: how long it will last, and how much of the traffic it hits.

Every example in this lesson uses the same pipeline. It reads 2,000 order events per second from a durable log, calls a tax service that can serve 3,000 requests per second, and writes each priced order to curated.orders, keyed on order_id.

Duration and scope

Scope is the newer idea. A record-scoped failure hits a single record because of something inside it: a malformed amount, a missing customer ID. A systemic failure hits every record because of something outside them all: an overloaded service, an expired credential, a bug in freshly deployed code. The same error can land on either side. A timeout on 1 call in 10,000 is record-scoped noise, while timeouts on every call are systemic.

ClassDurationScopeExampleResponse
Transient recordSecondsSingle recordA lone 503 or connection resetRetry in place with backoff and jitter
Transient systemicMinutes to hoursAll recordsTax service overloaded, 429 on every callOpen the breaker, leave records in the source log
Permanent recordForeverSingle record`tip_amount` holds the string 'abc'Write it to the DLQ and keep going
Permanent systemicUntil a human actsAll recordsExpired credential returning 401; a bug in the transformHalt the pipeline and page the owner

The bottom row is where pipelines most often go wrong. Record by record, a permanent systemic failure looks like a permanent record failure: the call returns 401 and the order cannot be priced. A pipeline that checks only the error type sends every order to the DLQ, and an hour later the DLQ holds 7.2 million good orders. All of them need the same fix, a new credential, and none of them needed a decision of its own.

The classifier is code, and it reads the failure rate

Classification belongs in a single function that every retry, breaker and DLQ decision calls, so the rules live in one place and can be tested. It takes 2 inputs: the error, and the share of recent records that failed with the same kind of error. For timeouts and overload responses, that share tracks the tax service. For parse errors, it tracks the payloads. That share is what promotes a record-scoped verdict to a systemic one.

from dataclasses import dataclass

TRANSIENT_STATUS = {429, 502, 503, 504}
AUTH_STATUS = {401, 403}
SYSTEMIC_RATE = 0.5


@dataclass
class Verdict:
    failure_class: str
    action: str


def classify(error, same_kind_failure_rate):
    status = getattr(error, "status", None)
    systemic = same_kind_failure_rate >= SYSTEMIC_RATE

    if isinstance(error, DuplicateKeyError):
        return Verdict("already_applied", "ack")
    if status in AUTH_STATUS:
        return Verdict("permanent_systemic", "halt_and_page")
    if status in TRANSIENT_STATUS or isinstance(error, (TimeoutError, ConnectionError)):
        if systemic:
            return Verdict("transient_systemic", "open_breaker")
        return Verdict("transient_record", "retry_with_backoff")
    if isinstance(error, (ValueError, KeyError)):
        if systemic:
            return Verdict("permanent_systemic", "halt_and_page")
        return Verdict("permanent_record", "send_to_dlq")
    return Verdict("unknown", "retry_then_dlq")

A duplicate-key error on an insert-if-absent write keyed on order_id and event_seq together means an earlier attempt already landed this exact event, so the record counts as a success and is acknowledged. A ValueError on half of all records points to a deployed bug or a producer format change, so the per-record verdict is overridden and the pipeline halts. An error the function has never seen gets a short retry and then the DLQ, and the count of unknown verdicts is tracked: when it climbs past a handful per hour, the classifier needs a new rule.

One table drives retry, breaker, queue and runbook

With classes defined, every mechanism downstream reads from the same table. That includes the runbook, the page an on-call engineer opens when an alert fires. It gets an entry per class instead of an entry per error message, which keeps it short enough to use at 3 a.m.

ClassRetryBreakerRecord goes toPagesRunbook first step
Transient recordUp to 4 attempts, exponential backoff, jitterCounts toward failure rateDLQ after attempt 4NoNone
Transient systemicPaused while breaker is openOpens at 50% failures over 30 sStays in the source logAfter 10 min openCheck the dependency's health and recent deploys
Permanent recordNoneNot countedDLQ with error signatureNoGroup the DLQ by signature
Permanent systemicNoneAll calls stoppedStays in the source logImmediatelyCheck the last deploy and credential expiry
Already appliedNoneNot countedAcknowledgedNoNone

A credential rotation at the tax service goes wrong, and from 14:00 every call from the order pipeline returns 401. Today the pipeline sends any record that fails 4 attempts to the DLQ. What should it do instead?

About this lesson

This section is part of the Failure Handling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.