Retry Policies and Backoff

Concepts covered: Retries and failure handling

retries and retry_delay give every failed try the same wait, and a fixed wait fits one kind of failure: a service that blinks for 20 seconds. A rate limit that lifts after 10 minutes, or a warehouse queue that drains over half an hour, outlasts a fixed policy. With 3 retries 1 minute apart, every retry lands inside the outage, so the task fails after roughly 3 minutes of waiting with 4 identical tracebacks. This section builds a policy that spaces its tries out, and gives up at once on the errors no wait can fix.

Doubling the wait

retry_exponential_backoff=True makes each wait longer than the one before it. Airflow computes the wait at the moment a try fails. The base is retry_delay multiplied by 2 raised to one less than the number of the try that failed, so the first wait is 1 base, the second 2 bases, the third 4. On top of the base it adds a jitter of between 0 and the base, taken from a hash of the DAG id and task id combined with the logical date and the try number. The hash is fixed for one task instance and differs across instances, so 200 mapped tasks that hit one rate limit together come back spread over a window rather than as a single wave that trips the limit again. max_retry_delay caps the result, jitter included.

from datetime import timedelta

from airflow.sdk import task


@task(
    retries=4,
    retry_delay=timedelta(minutes=5),
    retry_exponential_backoff=True,
    max_retry_delay=timedelta(minutes=30),
)
def pull_listings():
    ...

Add the last column and the fifth try starts between 65 and 90 minutes after the first failure, where a fixed 5 minute policy would have spent all 4 retries in 20 minutes. The wait is measured from the moment the try ended, so a try that runs for 20 minutes and then fails comes back 25 to 30 minutes after it began, which matters when a deadline downstream is close. In the grid the cell sits in up_for_retry for the whole wait: the scheduler stores the try's end time plus the wait as the earliest start and ignores the task until the clock reaches it. on_retry_callback fires as the cell enters that state, once per failed try, which is where a counter of how often each source needs a second try belongs.

Failures no wait will fix

A retry assumes the next try can end differently. A revoked API key, a file whose header names a column the warehouse lacks, a query with a typo in it: 5 tries of those cost up to 90 minutes, produce one traceback 5 times, and the person on call hears about it up to 90 minutes late. Airflow lets the task say that a wait would be wasted. Raising AirflowFailException moves the task instance straight to failed, however many retries remain. Any other exception moves it to up_for_retry while retries remain and to failed when they run out. The task body classifies the error it caught and raises the kind that fits.

from datetime import timedelta

import requests
from airflow.exceptions import AirflowFailException
from airflow.sdk import task
from etl import write_batch

LISTINGS_URL = "https://partner.example/v2/listings"


@task(retries=4, retry_delay=timedelta(minutes=5), retry_exponential_backoff=True)
def pull_listings():
    response = requests.get(LISTINGS_URL, timeout=30)
    if response.status_code in (400, 401, 403, 404):
        raise AirflowFailException(f"rejected: {response.status_code}")
    response.raise_for_status()
    write_batch(response.json())

Status codes 400 and 404 say the request itself was wrong, and so do 401 and 403; the next try would send the same request. A 429 is also a 4xx, but it says the request was fine and the caller should come back later, so the code leaves it off the list. It falls through to raise_for_status() with the 5xx answers, which say the server was wrong, and raises an ordinary exception; a timeout raises one from requests.get itself, and in each case the backoff policy takes over. For a source that has nothing to give today, AirflowSkipException ends the try as skipped, and the try is neither retried nor failed. A try that overruns execution_timeout is killed and counted as a failed try, so a task with 4 retries and a 10 minute timeout can hold a worker for 50 minutes of running time before it fails for good.

Do

• Set backoff and a cap together so the last wait still fits before the run's deadline
• Raise AirflowFailException on 4xx, auth and schema errors the next try would repeat
• Keep retries in default_args and override them on the tasks that call flaky services

Avoid

• Retry a task that writes without a way to redo the write cleanly
• Lean on retries to cover a memory limit the task exceeds on every try
• Set retry_delay in seconds against a rate limit measured in minutes

About this lesson

This section is part of the Task Lifecycle: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.