Deadline Alerts

Concepts covered: Retries and failure handling

A failed run calls on_failure_callback the moment it fails. A run that is still going at 07:00, when the pricing dashboard refreshes from its output, calls nothing, because nothing has failed: pull_listings hit its rate limit at 03:10, backed off 4 times, and is on its fifth try with every other cell green or waiting. The people at the dashboard find stale prices and ask the data team at 09:00, 2 hours after the point at which a page could have changed the outcome. A deadline alert is Airflow's answer for that run: a time the run must be finished by, and a callback that runs when the clock passes it with the run still open.

from datetime import timedelta

from airflow.sdk import dag
from airflow.sdk.definitions.deadline import AsyncCallback, DeadlineAlert, DeadlineReference
from airflow.timetables.trigger import CronTriggerTimetable


@dag(
    schedule=CronTriggerTimetable("0 2 * * *", timezone="UTC"),
    catchup=False,
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_LOGICAL_DATE,
        interval=timedelta(hours=5),
        callback=AsyncCallback(
            "pricing.alerts.page_oncall",
            kwargs={"channel": "#pricing-oncall"},
        ),
    ),
)
def nightly_pricing():
    ...


nightly_pricing()

The deadline is a reference plus an interval, computed once when the run is created and stored with it. DAGRUN_LOGICAL_DATE anchors to the run's logical date, and with CronTriggerTimetable that is the 02:00 the run fired at, so the deadline is 07:00 the same morning for every scheduled run. With CronDataIntervalTimetable the logical date would be the previous day's 02:00, and the same 5 hours would name a deadline already past when the run is created, so the alert fires as the run starts; the timetable lesson's distinction decides whether the alert means anything. DAGRUN_QUEUED_AT anchors to the moment the run was created, which measures how long the run took regardless of when it was due, and FIXED_DATETIME names one instant for a run that must land before a specific meeting.

The scheduler checks pending deadlines on each loop, and a deadline whose run is still open when the clock passes it fires once. AsyncCallback names an async def in a module the triggerer can import, and the triggerer runs it with the kwargs you set alongside details of the run that missed, so the page can link to the grid. Running it in the triggerer matters because a late run is often late for lack of workers; a callback that needed a worker slot would queue behind the tasks it is reporting on. The alert fires whether the run then succeeds at 07:25 or fails at 09:00, since what it measures is lateness, and a run that fails before 07:00 reaches a terminal state and never fires it.

dagrun_timeout

• A duration measured from the run's start
• Fails the run when exceeded, killing its running tasks
• Fires on_failure_callback through the failure
• Answers: has this run gone on too long?

DeadlineAlert

• An instant computed from a reference plus an interval
• Changes nothing about the run; it keeps going
• Runs its own callback in the triggerer, once
• Answers: will the people waiting get it on time?

Set the interval from the consumer's clock, then subtract the retry policy's worst case. A deadline at 07:00 with pull_listings able to wait 90 minutes across its retries means a first failure after 05:30 cannot finish in time, which is a fact to carry back to the backoff section: cap the waits so the last retry still fits. A DAG whose runs are late on most nights has a capacity problem, and the alert will report it every morning until the platform has room for that DAG; the alert only reports the lateness, and the fix lies in the platform's capacity.

About this lesson

This section is part of the Task Lifecycle: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.