Retry Budgets: Max, Delay, Jitter

Concepts covered: Retry & Failure Handling

4 limits around a retry loop

The beginner lesson gave retries a delay formula: wait longer after each failure. A retry budget is the set of limits around that formula, and each limit stops a different quantity from growing without bound. The payments loader's warehouse write uses the budget below, plus a timeout of 2 s on every attempt so a hung call cannot stretch a single attempt into minutes.

LimitWhat it boundsLoader setting
Max attemptsCalls one message makes to the warehouse6
Base delay and capThe length of any single wait0.5 s, doubling, capped at 8 s
Elapsed deadlineTotal time a message can hold a worker20 s
JitterHow many workers retry at the same instantFull: a random wait from 0 to the computed delay

Walking the worst case

In the worst case every attempt hangs until its 2 s timeout. With jitter switched off so the arithmetic stays visible, the schedule runs like this.

AttemptStartsFailsWait before next
10.0 s2.0 s0.5 s
22.5 s4.5 s1 s
35.5 s7.5 s2 s
49.5 s11.5 s4 s
515.5 s17.5 s8 s
6Would start at 25.5 sWould end at 27.5 sRefused: past the 20 s deadline

The loader refuses to start an attempt that couldn't finish inside the deadline. Attempt 6 would run from 25.5 s to 27.5 s, so the loader stops at 17.5 s and hands the failure to the next layer. Max attempts allowed 6; the deadline allowed 5. Having both is deliberate: max attempts limits the load on the warehouse, and the deadline limits how stale the pipeline gets.

import random
import time


def call_with_budget(operation, max_attempts=6, base=0.5, cap=8.0,
                     deadline=20.0, attempt_timeout=2.0):
    start = time.monotonic()
    for attempt in range(1, max_attempts + 1):
        try:
            return operation(timeout=attempt_timeout)
        except TransientError:
            if attempt == max_attempts:
                raise
            delay = random.uniform(0, min(cap, base * 2 ** (attempt - 1)))
            elapsed = time.monotonic() - start
            if elapsed + delay + attempt_timeout > deadline:
                raise
            time.sleep(delay)

Only TransientError is caught. A permanent error, such as the 12,50 parse failure, passes straight through on the first attempt, which is how it reached the DLQ with attempts: 1.

A loader's budget is 10 max attempts, a 1 s base delay that doubles, a 4 s cap, a 5 s timeout per attempt, and no elapsed deadline. If every attempt hangs until its timeout, how long can a single message hold a worker?

Jitter sized to what an attempt costs

Full jitter replaces each computed delay with a random wait between 0 and that delay, so 200 workers that failed in the same second spread their retries across a window. The window has to be wide enough that the retries arrive no faster than the downstream can serve them, and that depends on how long each attempt holds a connection.

The minimum window is workers * seconds_per_attempt / concurrent_slots. Take a downstream that serves 20 requests at once. A currency lookup that holds a connection for 50 ms needs 200 * 0.05 / 20 = 0.5 s of spread. A bulk load that holds a warehouse slot for 30 s needs 200 * 30 / 20 = 300 s. The 0.5 s base delay that spreads the lookups well would start all 200 bulk loads within the same half-second, 10 times what the warehouse accepts.

When the warehouse fails for every message at once, each message spends its whole budget and the warehouse gets several times its normal traffic while it's weakest. A per-worker retry ratio closes the gap: each worker drops any retry that would push its retries above 10% of its calls in the last minute, which bounds fleet load because every worker applies it. A dropped retry leaves the message uncommitted for the next pass.

After a network blip, 120 backfill workers fail at the same moment. Each retry runs a 10 s query against a database that allows 40 concurrent queries. What is the smallest jitter window that keeps around 40 queries running at once?

About this lesson

This section is part of the Failure Handling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.