Circuit Breakers Stop the Hammer

When retries become the outage

A retry budget bets that the next attempt might work. During a long outage that bet loses every time. At 18:10 the currency service that enrich_fx calls to convert EUR fares to USD stops answering for 10 minutes. The enricher uses the same budget as the loader, so each worker spends 5 attempts and 17.5 s on a message, leaves its offset uncommitted, and starts over on the same message. No payment gets through, and the service absorbs a steady flood of calls while it tries to restart.

A circuit breaker sits between a caller and a single dependency, watches the outcomes of recent calls, and stops calling once failures cross a threshold. While it's open, every call fails at once with CircuitOpen. That failure takes microseconds and sends nothing over the network, so the struggling service gets quiet in which to recover.

Closed, open, half-open

StateWhat a call doesNext stateWhen
ClosedGoes through; its outcome joins a window of the last 20 callsOpenAt least 10 of the last 20 calls failed
OpenFails at once with `CircuitOpen`, no network callHalf-openThe 30 s cooldown has passed
Half-openLets 3 trial calls through; the rest fail at onceClosedAll 3 trials succeed
Half-openSame trial ruleOpenAny trial fails; the 30 s cooldown restarts

Each cooldown ends with 3 trial calls, a load any restarting service can take. When the service returns at 18:20, all 3 calls in the next probe succeed and the breaker closes. Traffic resumes within 30 s of the recovery.

Why each number is there

A breaker opens at 14:00:00 with a 30 s cooldown and 3 trial calls. At 14:00:31 the first trial succeeds and the second times out. What happens next?

What the pipeline does with CircuitOpen

The code that catches CircuitOpen decides what happens next. The error says nothing about the message, so it is classified as transient and the message never goes to the DLQ. The enricher stops polling and leaves its committed offset where it is. After the cooldown, allow_request lets it take up to 3 messages from that offset as the half-open trials. Once all 3 succeed, the breaker closes; consumption picks up from that offset, so no payment is lost or skipped. A single failed trial pauses the enricher again.

def enrich(message, breaker, fx_client):
    if not breaker.allow_request():
        raise CircuitOpen("fx_rates")
    try:
        rate = fx_client.get_rate(message["currency"], timeout=2.0)
    except TransientError:
        breaker.record_failure()
        raise
    breaker.record_success()
    return {**message, "amount_usd": message["amount"] * rate}

This function runs inside call_with_budget, so every retry attempt counts toward the failure rate. CircuitOpen is deliberately a separate error from TransientError: the retry loop does not catch it, so an open breaker ends the loop on the spot instead of sleeping through 5 attempts that are certain to fail.

Retry budget
  • Scope: a single call and its attempts
  • Expects the next attempt to work
  • Adds load: up to 6 calls per message
  • Fits blips that last seconds
Circuit breaker
  • Scope: every call to a dependency
  • Expects failure until a trial succeeds
  • Removes load: 0 calls while open
  • Fits outages that last minutes

A teammate changes the handler so that CircuitOpen sends the message to trip_payments_dlq, since a fast failure is still a failure. What does the next 10-minute currency outage do?

About this lesson

This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.