Circuit Breakers Stop the Hammer
When retries become the outage
A retry budget bets that the next attempt might work. During a long outage that bet loses every time. At 18:10 the currency service that enrich_fx calls to convert EUR fares to USD stops answering for 10 minutes. The enricher uses the same budget as the loader, so each worker spends 5 attempts and 17.5 s on a message, leaves its offset uncommitted, and starts over on the same message. No payment gets through, and the service absorbs a steady flood of calls while it tries to restart.
A circuit breaker sits between a caller and a single dependency, watches the outcomes of recent calls, and stops calling once failures cross a threshold. While it's open, every call fails at once with CircuitOpen. That failure takes microseconds and sends nothing over the network, so the struggling service gets quiet in which to recover.
- 57 per s
- 0
Closed, open, half-open
| State | What a call does | Next state | When |
|---|---|---|---|
| Closed | Goes through; its outcome joins a window of the last 20 calls | Open | At least 10 of the last 20 calls failed |
| Open | Fails at once with `CircuitOpen`, no network call | Half-open | The 30 s cooldown has passed |
| Half-open | Lets 3 trial calls through; the rest fail at once | Closed | All 3 trials succeed |
| Half-open | Same trial rule | Open | Any trial fails; the 30 s cooldown restarts |
Each cooldown ends with 3 trial calls, a load any restarting service can take. When the service returns at 18:20, all 3 calls in the next probe succeed and the breaker closes. Traffic resumes within 30 s of the recovery.
Why each number is there
- A window of 20 calls: Without a minimum, 1 failure in the first 2 calls reads as 50%.
- A 50% failure rate: A few slow calls leave it closed; a dead service trips it in seconds.
- A 30 s cooldown: Long enough for a restart to finish, short enough to notice recovery within a minute.
- Trials of 3 calls: A single success could be luck; 3 in a row is evidence, and far short of a stampede.
A breaker opens at 14:00:00 with a 30 s cooldown and 3 trial calls. At 14:00:31 the first trial succeeds and the second times out. What happens next?
- It closes, because a trial succeeded
- It stays half-open until the third trial returns
- It stays open until an operator resets it
- It opens again, with new trials about 30 s later
What the pipeline does with CircuitOpen
The code that catches CircuitOpen decides what happens next. The error says nothing about the message, so it is classified as transient and the message never goes to the DLQ. The enricher stops polling and leaves its committed offset where it is. After the cooldown, allow_request lets it take up to 3 messages from that offset as the half-open trials. Once all 3 succeed, the breaker closes; consumption picks up from that offset, so no payment is lost or skipped. A single failed trial pauses the enricher again.
def enrich(message, breaker, fx_client):
if not breaker.allow_request():
raise CircuitOpen("fx_rates")
try:
rate = fx_client.get_rate(message["currency"], timeout=2.0)
except TransientError:
breaker.record_failure()
raise
breaker.record_success()
return {**message, "amount_usd": message["amount"] * rate}
This function runs inside call_with_budget, so every retry attempt counts toward the failure rate. CircuitOpen is deliberately a separate error from TransientError: the retry loop does not catch it, so an open breaker ends the loop on the spot instead of sleeping through 5 attempts that are certain to fail.
- Retry budget
- Scope: a single call and its attempts
- Expects the next attempt to work
- Adds load: up to 6 calls per message
- Fits blips that last seconds
- Circuit breaker
- Scope: every call to a dependency
- Expects failure until a trial succeeds
- Removes load: 0 calls while open
- Fits outages that last minutes
A teammate changes the handler so that CircuitOpen sends the message to trip_payments_dlq, since a fast failure is still a failure. What does the next 10-minute currency outage do?
- Nothing changes: the breaker keeps calls off the service and the DLQ stays empty
- It fills the DLQ with valid payments for 10 minutes, each needing a replay
- It keeps the breaker closed, since dead-lettered calls skip the failure count
- It pauses the consumer at its offset as before, and only the DLQ alert changes
About this lesson
This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.