Timeouts and Hung Tasks

Concepts covered: Retries and failure handling

A task that never returns stays in running. Its process holds the worker it was given, it keeps reporting that it is alive, and the scheduler has no reason to interfere: as far as Airflow can see, a 9-hour try is a long try. The run stays in running with it, and the log stops after the last line printed. execution_timeout on the task and dagrun_timeout on the DAG put a clock on that.

from datetime import datetime, timedelta

from airflow.sdk import dag, task
from sessions_api import fetch_sessions


@dag(
    schedule="@hourly",
    start_date=datetime(2026, 11, 1),
    catchup=False,
    dagrun_timeout=timedelta(minutes=50),
)
def hourly_sessions():
    @task(
        execution_timeout=timedelta(minutes=15),
        retries=1,
        retry_delay=timedelta(minutes=2),
    )
    def pull_sessions():
        fetch_sessions()

    @task(execution_timeout=timedelta(minutes=10))
    def write_sessions():
        print("sessions written")

    pull_sessions() >> write_sessions()


hourly_sessions()

execution_timeout is a timedelta measured per try from the moment the try starts running. When it elapses, the task runner raises AirflowTaskTimeout inside the task's process, and the try ends as a failure. Retries and callbacks treat it like any other failure: with retries=1, the first timed-out try goes to up_for_retry and on_retry_callback fires if one is set; after retry_delay a second try starts with its own fresh 15 minutes; if that one times out as well, the task reaches failed and on_failure_callback fires. write_sessions becomes upstream_failed and the run is marked failed at 01:32, 32 minutes after it began, rather than never.

from datetime import datetime, timedelta
from airflow.sdk import dag, task

@dag(schedule=timedelta(hours=1),
     start_date=datetime(2026, 11, 2))
def hourly_sessions():
    @task(retries=1,
          retry_delay=timedelta(minutes=2),
          execution_timeout=timedelta(minutes=15))
    def pull_sessions(): ...
    @task
    def write_sessions(): ...
    pull_sessions() >> write_sessions()
hourly_sessions()

The run's own clock

dagrun_timeout is measured from the run's start, the moment it moved to running, which for a catch-up run is long after its logical date. When a run has been running longer than that, the scheduler marks it failed, sets every unfinished task in it to skipped, and fires the DAG's on_failure_callback; a task that was mid-try sees its state change on its next heartbeat and its process is stopped. The parameter is documented as enforced for scheduled runs, so a run you trigger by hand keeps going. In hourly_sessions the 50-minute limit sits behind the task limits as a backstop: the worst case for pull_sessions alone is 32 minutes, write_sessions adds at most 10, so a run that meets the backstop has something wrong that the task limits did not cover.

Without either setting, a hung task shows up in the grid as a cell that has been light green for hours while its neighbours in older runs took 4 minutes, with a log whose last line is older than the run. Marking it failed from the grid stops the process and ends the try as failed, after which the run's state recalculates and the usual callbacks fire; clearing it starts a fresh try instead. Both are a person doing by hand at 09:00 what a 15-minute limit would have done at 01:15.

Do

• Give every task that calls out of Airflow an execution_timeout a few times its normal duration
• Set retries alongside the timeout, so a slow remote gets a second chance before the task fails
• Set dagrun_timeout longer than the sum of the task limits, so it only fires on what they missed

Avoid

• Set execution_timeout shorter than a healthy try on a slow day, or the timeout becomes the failure
• Expect dagrun_timeout to stop a run you triggered by hand
• Read a long-running cell as progress without opening its log

About this lesson

This section is part of the Task Lifecycle: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.