Callbacks on Failure

Concepts covered: Retries and failure handling

A callback is a Python function that Airflow calls with the task's context at a fixed moment in a try's life. A task has 5 of them: on_execute_callback as a try starts, on_success_callback when a try ends in success, on_retry_callback when a try fails and a retry remains, on_failure_callback when the last allowed try fails and the task reaches failed, and on_skipped_callback when the task's own code raises AirflowSkipException. Each accepts 1 function or a list of them, and each runs in the task's own process on the worker, right after the try ends. A task that never ran a process fires none of them.

from datetime import datetime, timedelta

from airflow.sdk import dag, task


def page_oncall(context):
    ti = context["task_instance"]
    print(f"{ti.dag_id}.{ti.task_id} failed on try {ti.try_number}: {context['exception']}")


def note_retry(context):
    ti = context["task_instance"]
    print(f"{ti.task_id} try {ti.try_number} failed, retry queued")


def summarize_run(context):
    run = context["dag_run"]
    print(f"run {run.run_id} of {run.dag_id} failed")


@dag(
    schedule="@daily",
    start_date=datetime(2026, 11, 1),
    catchup=False,
    on_failure_callback=summarize_run,
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
        "on_retry_callback": note_retry,
        "on_failure_callback": page_oncall,
    },
)
def payments_rollup():
    @task
    def pull_ledger():
        raise ConnectionError("ledger API 503")

    @task
    def rollup():
        print("payments rolled up")

    pull_ledger() >> rollup()


payments_rollup()

context['task_instance'] is the instance that ran, carrying its dag_id, task_id and try_number, and context['exception'] is the exception the try raised, so the page reads payments_rollup.pull_ledger failed on try 3: ledger API 503. default_args hands every task in the DAG the same retries and the same callbacks, so a task added next month is paged without anyone remembering to wire it. The on_failure_callback passed to @dag itself is a different hook: it fires once when the run is marked failed, receives the run through context['dag_run'] rather than a task, and is executed by the dag processor rather than by a worker, so it fires even when the failure was an upstream_failed leaf that never ran.

A night with 3 failed tries

The task made 3 tries and fired 3 callbacks, and the run fired 1. note_retry fires twice because 2 retries were allowed, page_oncall fires once when the third try exhausts them, and summarize_run fires when the run is marked failed. rollup was set to upstream_failed by the scheduler and, since no try of it ever ran, fires nothing, which is the behavior you want: 1 page for the root cause rather than 1 per orange cell. A task the scheduler skips through a trigger rule or a branch fires nothing for the same reason.

A callback is the right place for a side effect that needs the failure's details: a page with the task id, a message in a chat channel, a row in an incidents table. It is the wrong place for work the pipeline depends on, because an exception inside the callback is written to the task's log and otherwise ignored, so the task's state stays where it was and nothing downstream notices. A cleanup that must happen belongs in an all_done task, where its own cell shows whether it ran.

About this lesson

This section is part of the Task Lifecycle: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.