Retry Budgets: Max, Delay, Jitter
Concepts covered: Retry & Failure Handling
4 limits around a retry loop
The beginner lesson gave retries a delay formula: wait longer after each failure. A retry budget is the set of limits around that formula, and each limit stops a different quantity from growing without bound. The payments loader's warehouse write uses the budget below, plus a timeout of 2 s on every attempt so a hung call cannot stretch a single attempt into minutes.
| Limit | What it bounds | Loader setting |
|---|---|---|
| Max attempts | Calls one message makes to the warehouse | 6 |
| Base delay and cap | The length of any single wait | 0.5 s, doubling, capped at 8 s |
| Elapsed deadline | Total time a message can hold a worker | 20 s |
| Jitter | How many workers retry at the same instant | Full: a random wait from 0 to the computed delay |
Walking the worst case
In the worst case every attempt hangs until its 2 s timeout. With jitter switched off so the arithmetic stays visible, the schedule runs like this.
| Attempt | Starts | Fails | Wait before next |
|---|---|---|---|
| 1 | 0.0 s | 2.0 s | 0.5 s |
| 2 | 2.5 s | 4.5 s | 1 s |
| 3 | 5.5 s | 7.5 s | 2 s |
| 4 | 9.5 s | 11.5 s | 4 s |
| 5 | 15.5 s | 17.5 s | 8 s |
| 6 | Would start at 25.5 s | Would end at 27.5 s | Refused: past the 20 s deadline |
The loader refuses to start an attempt that couldn't finish inside the deadline. Attempt 6 would run from 25.5 s to 27.5 s, so the loader stops at 17.5 s and hands the failure to the next layer. Max attempts allowed 6; the deadline allowed 5. Having both is deliberate: max attempts limits the load on the warehouse, and the deadline limits how stale the pipeline gets.
import random
import time
def call_with_budget(operation, max_attempts=6, base=0.5, cap=8.0,
deadline=20.0, attempt_timeout=2.0):
start = time.monotonic()
for attempt in range(1, max_attempts + 1):
try:
return operation(timeout=attempt_timeout)
except TransientError:
if attempt == max_attempts:
raise
delay = random.uniform(0, min(cap, base * 2 ** (attempt - 1)))
elapsed = time.monotonic() - start
if elapsed + delay + attempt_timeout > deadline:
raise
time.sleep(delay)
Only TransientError is caught. A permanent error, such as the 12,50 parse failure, passes straight through on the first attempt, which is how it reached the DLQ with attempts: 1.
A loader's budget is 10 max attempts, a 1 s base delay that doubles, a 4 s cap, a 5 s timeout per attempt, and no elapsed deadline. If every attempt hangs until its timeout, how long can a single message hold a worker?
- A total of 31 s, the sum of the 9 waits
- A total of 50 s, 10 attempts of 5 s each
- A total of 81 s: 50 s of attempts plus 31 s of waits
- No limit at all, because nothing sets a deadline
Jitter sized to what an attempt costs
Full jitter replaces each computed delay with a random wait between 0 and that delay, so 200 workers that failed in the same second spread their retries across a window. The window has to be wide enough that the retries arrive no faster than the downstream can serve them, and that depends on how long each attempt holds a connection.
The minimum window is workers * seconds_per_attempt / concurrent_slots. Take a downstream that serves 20 requests at once. A currency lookup that holds a connection for 50 ms needs 200 * 0.05 / 20 = 0.5 s of spread. A bulk load that holds a warehouse slot for 30 s needs 200 * 30 / 20 = 300 s. The 0.5 s base delay that spreads the lookups well would start all 200 bulk loads within the same half-second, 10 times what the warehouse accepts.
When the warehouse fails for every message at once, each message spends its whole budget and the warehouse gets several times its normal traffic while it's weakest. A per-worker retry ratio closes the gap: each worker drops any retry that would push its retries above 10% of its calls in the last minute, which bounds fleet load because every worker applies it. A dropped retry leaves the message uncommitted for the next pass.
- 6x
- 1.1x
After a network blip, 120 backfill workers fail at the same moment. Each retry runs a 10 s query against a database that allows 40 concurrent queries. What is the smallest jitter window that keeps around 40 queries running at once?
- A window of 30 s, so each slot runs 3 retries in turn
- A window of 3 s, the worker count divided by the slots
- A window of 0.25 s, the query time divided by the slots
- A window of 400 s, the slot count times the query time
About this lesson
This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.