Backfilling a Date Range
Concepts covered: Catchup and backfill
Catchup fills windows the scheduler missed while a DAG was paused or did not yet exist, and it fills each one once. A backfill reruns windows on purpose: you pick a date range, runs are created for it whether they ran before, and it works with catchup on or off. In 3.3 the scheduler owns backfills. You open one from the DAG's page or with airflow backfill create, the request is stored, and the scheduler creates the runs at the pace you set, beside the live schedule rather than in place of it.
The command form carries the same choices as the dialog: airflow backfill create --dag-id orders_daily --from-date 2026-10-01 --to-date 2026-10-04 --reprocess-behavior none --max-active-runs 2.
Which runs it creates
The scheduler walks the timetable's schedule points from the from date forward and creates a run for every point whose window has already closed, provided its logical date falls on or between the 2 dates you gave. Both dates are instants: a bare date is midnight UTC at the start of that day. Take the 06:00 UTC DAG from section 0 and the backfill above. The schedule points are 06:00 on each day. The points from the 1st of October through the 3rd fall inside the range. The point at 06:00 on the 4th sits 6 hours past the to date and is left out, so the last window the backfill covers ends at 06:00 on the 4th. Each created run has a run type of backfill and an id of backfill__ followed by its logical date, and each task in it reads the window of its own point, which is why section 0 insisted on the window fields.
| schedule point | inside 2026-10-01 to 2026-10-04? | window | run created |
|---|---|---|---|
| 2026-10-01 06:00 | yes | 10-01 06:00 to 10-02 06:00 | backfill__2026-10-01T06:00:00+00:00 |
| 2026-10-02 06:00 | yes | 10-02 06:00 to 10-03 06:00 | backfill__2026-10-02T06:00:00+00:00 |
| 2026-10-03 06:00 | yes | 10-03 06:00 to 10-04 06:00 | backfill__2026-10-03T06:00:00+00:00 |
| 2026-10-04 06:00 | no, 6 hours past the to date | 10-04 06:00 to 10-05 06:00 | none |
Dates that already have a run
The reprocess behavior decides what happens when a point inside the range already has a run. none, the default, creates nothing for that point and records it as already existing, so the backfill touches only dates with no run. failed clears and reruns the points whose existing run failed and leaves successful ones alone. completed goes further and reruns the successful ones too, so every finished run in the range is cleared and run again. A run that is still running is left alone under every setting. Runs are created from the oldest point forward, and --run-backwards reverses that order, which helps when the newest dates matter most to whoever is waiting on them.
| existing run for the point | none | failed | completed |
|---|---|---|---|
| no run | created | created | created |
| failed | left alone | cleared and rerun | cleared and rerun |
| succeeded | left alone | left alone | cleared and rerun |
| still running | left alone | left alone | left alone |
--max-active-runs caps how many of the backfill's runs can be in flight at once; the rest wait as earlier ones finish, so a 90-day backfill with a cap of 2 occupies 2 slots of worker capacity and takes 45 rounds. --dry-run lists the logical dates the backfill would create without creating anything, which is the check to run whenever the to date is a bare date and the cron fires later than midnight. To pause a backfill in progress, pass its id to airflow backfill pause; airflow backfill unpause resumes it and airflow backfill cancel cancels it. airflow backfill list shows the backfills that exist and their state.
- Do
- Run --dry-run first and read the logical dates it prints
- Set the to date at or after the last logical date you want, such as 06:00 on that day
- Cap active runs below what the source system tolerates
- Choose failed over completed when successful runs are still correct
- Avoid
- Assume a bare to date includes that day's 06:00 point
- Backfill a task that reads today's date instead of its window
- Leave the reprocess behavior at none when the data was restated
- Start a wide backfill without checking the live schedule shares the workers
About this lesson
This section is part of the Scheduling: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.