Retries and Retry Delay

Concepts covered: Retries and failure handling

Picture the nightly_orders run at 00:08. extract took 2 minutes and transform took 3, both ended success, and load has been running for 3 minutes when the warehouse drops its connection and the task raises. The 2 cells above it did their work, and the failure says nothing about their output. If the unit of a retry were the whole run, the fix would redo 5 minutes of finished work to reach the 1 task that broke. So Airflow retries the task instance alone: the 2 green cells stay green, and only load gets another try.

2 settings on a task control this. retries is how many extra tries the task may take after the first one raises, and it defaults to 0. retry_delay is how long the scheduler waits between a try that raised and the next one, a timedelta that defaults to 300 seconds. With retries=2 a task can run 3 times in total; the count names the extra tries, never the first one.

Between one try and the next

When load raises, the worker reports the failure along with the try number, 1, and the report is checked against the task's retries, 2. Tries remain, so the cell is recorded as up_for_retry instead of failed, along with the time of the raise. For the next 5 minutes the cell sits in up_for_retry: no worker holds it, nothing of the task is executing, and the grid shows the cell in the waiting colour. The scheduler's part is the wait: once retry_delay has elapsed it moves the cell back to scheduled, then queued, and a worker takes it to running as try number 2. In the night below, try 2 returns, the cell ends success, and the run ends success about 8 minutes later than it would have without the outage. Had try 2 raised as well, the same loop would run once more, and try 3 would be the last one retries=2 allows: a raise there is recorded as failed, which is final.

import datetime as dt
from airflow.sdk import task, DAG

with DAG(
    dag_id="nightly_orders",
    start_date=dt.datetime(2026, 3, 1),
    schedule="@daily",
    catchup=False,
):
    @task
    def extract() -> None: ...
    @task
    def transform() -> None: ...
    @task(
        retries=2,
        retry_delay=dt.timedelta(minutes=5),
    )
    def load() -> None: ...
    extract() >> transform() >> load()

Set the pair once on the DAG with default_args={"retries": 1, "retry_delay": timedelta(minutes=2)} and every task inherits it; a task that passes its own retries overrides the inherited value for itself only. The order Airflow reads is the task's own argument, then default_args, then its built-in default. In the DAG above, extract and transform carry the built-in 0, so a raise in either of them is final on the first try.

Choosing the numbers

The worst case a retry setting creates is the time until the cell is finally failed: every try's running time plus every wait. load runs about 3 minutes before it raises, so retries=2 with a 5 minute delay means 3 tries of 3 minutes and 2 waits of 5 minutes, 19 minutes from the first start to a red cell. Pick the delay from the outage you are covering: a warehouse that drops connections for a minute at a time is served by a short delay, and a service that restarts over 10 minutes needs a delay longer than that or every try lands inside the same outage.

Do

• Give tasks that cross a network a retry; a dropped connection heals on its own.
• Make retry_delay longer than the outage you expect, or every try fails inside it.
• Set the pair in default_args once and override on the few tasks that differ.

Avoid

• Retry a task whose failure is a code bug; every try fails the same way.
• Read retries=3 as 3 tries; it is 4.
• Leave a critical task at the default of 0 and discover it on the first bad night.

About this lesson

This section is part of the Task Lifecycle: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.