Catchup On and Off

Concepts covered: Catchup and backfill

Windows that closed while nobody was watching

Unpause a DAG later than its start_date and some windows have already closed with no run created for them. catchup decides what happens to them. With catchup=True, Airflow creates 1 run for every closed window, oldest first, the moment the DAG is unpaused, and the runs execute side by side. With catchup=False, only the most recently closed window gets a run; the older windows never run, and nothing marks them in the grid.

Take daily_refunds, which runs at 00:00 every day from a start_date of 3 November 2026, unpaused at 06:30 on 5 November. 2 windows have closed by then: 3 to 4 November and 4 to 5 November. With catch-up on, both runs are created at 06:30 and start together. With it off, Airflow creates only the 2026-11-04 run, and 3 November is never counted. From midnight on 6 November the 2 settings behave the same, 1 run as each window closes.

from airflow.sdk import task, dag
import pendulum
import airflow.timetables.interval as iv

@dag(
    dag_id="daily_refunds",
    schedule=iv.CronDataIntervalTimetable(
        "0 0 * * *", timezone="UTC"),
    start_date=pendulum.datetime(
        2026, 11, 3, tz="UTC"),
    catchup=True,
)
def daily_refunds():
    @task
    def total_refunds():
        ...
    total_refunds()

daily_refunds()

The animation shows the 06:30 moment from both sides. The 2 variants have the same file apart from 1 line, and that line decides whether 3 November exists in the table the DAG writes. Airflow 3 defaults catchup to False, so a DAG that says nothing about it skips to the latest window; write the setting out either way so a reader of the file knows what will happen the day it is unpaused.

In the grid, catch-up runs appear as a block of columns created within the same minute, their names stepping back a day at a time, with the newest at the right. Each is handed its own window, so the run named 3 November reads 3 November's rows even though it executes on the 5th. Because runs are named by window, a run can execute days late and still read the rows it was meant to read.

The same rule applies every time a DAG is unpaused, with the gap measured from its last scheduled run instead of from start_date. Pause daily_refunds for a week of maintenance and unpause it with catchup=True, and 7 runs are created at once; with catchup=False, 1 run is created and the other 6 days are skipped for good.

Counting the backlog before unpausing

The number of catch-up runs is the number of closed windows between start_date and now, and that number grows fast with a short period. Catch-up runs begin several at a time, so a DAG with a long history behind it can occupy the workers for hours while the current day's run waits in the queue behind them. Count before you unpause.

• 17,520
• 730
• 104

Choosing a setting

Turn catch-up on when

• each window's rows are kept, 1 row or 1 partition per day, and a missing day is a hole
• the DAG rebuilds history and every window must exist exactly once
• start_date is recent and the backlog is a handful of runs

Leave it off when

• the run reads the current state of a source and older windows would read the same rows
• the backlog is months of windows nobody asked for
• the DAG is new and 1 run today is enough, with history handled another way

About this lesson

This section is part of the Scheduling: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.