Apache Airflow Schedule Interval

The logical date is the new name for what Airflow 2 called the execution date, and the name was always misleading, because it isn't when the run executes. The Dag run documentation says the logical date "denotes the start of the data interval, not when the Dag is actually executed". Airflow 3.0 removed the execution_date context variable, along with next_execution_date, prev_execution_date, next_ds and prev_ds, and tells you to read dag_run.logical_date instead.

The Airflow 3 upgrade guide lists those removals and warns that leaving them in a Dag will cause Dag errors. Manual runs changed too: Airflow no longer accepts a logical date in the future, and logical_date=None triggers a run with the current timestamp.

For a manually triggered run, don't assume the data interval follows from the logical date you supplied. If a Dag needs the date the user chose, read logical_date explicitly. If it needs the range of data to process, read the data interval.

A string such as 0 0 * * * is shorthand for a timetable, and which one it becomes is set by the create_cron_data_intervals option in the [scheduler] section. It was added in Airflow 2.9.0. Its default is False, which means a bare cron string becomes a CronTriggerTimetable. Set to True, it becomes a CronDataIntervalTimetable. The Airflow 3 upgrade guide notes the default change, and says it only affects Dags that pass a bare cron string, so a Dag that names its timetable is unaffected.

The difference shows in the first run. The timetables documentation enables a daily Dag at 15:00 on January 31. The trigger timetable creates its first run at midnight on February 1, while the data interval timetable creates a run immediately, with a run id timestamp of midnight on January 31. If the Dag is paused on January 31 and enabled again on February 2, the trigger timetable skips the runs for February 1 and 2, and the data interval timetable skips only February 1 and triggers February 2 at once.

If you want a trigger timetable that still carries a real data interval, pass interval. The documented example, 0 18 * * 5 with timedelta(days=4, hours=9), runs on Fridays at 18:00 and covers Monday 09:00 to that Friday, which is 4 x 24 + 9 = 105 hours. For several cron expressions on 1 Dag, MultipleCronTriggerTimetable runs whenever any of them matches.

An Airflow cron expression has 5 fields: minute, hour, day of month, month and day of week. Airflow parses it with croniter, so extended forms work, for instance 0 0 * * MON#1 for midnight on the first Monday of the month. The expression */30 * * * * fires every 30 minutes on the clock, which is different from a 30 minute timedelta, and Airflow also accepts None to never schedule and @once to schedule a single run.

Presets are named cron expressions. Their documented descriptions say "at midnight (24:00)" for the daily, weekly, monthly, quarterly and yearly presets, and "at the end of the hour" for @hourly. Read these with the timetable in mind: under a data interval timetable the run for a day starts after that day ends.

A preset or cron string is the shortest way to write a schedule, but it hides the timetable choice. A Dag that depends on the data interval should name its timetable instead of relying on the create_cron_data_intervals setting of whoever deploys it.

By default the scheduler creates a run only for the latest interval. Airflow 3.0 made catchup_by_default False, so a Dag that doesn't set catchup will no longer backfill missed intervals. With catchup=True, the scheduler starts a run for every past interval since the start date that hasn't run, and it does the same for the gap when a paused Dag is switched on again. Several of those runs can overlap, bounded by max_active_runs_per_dag, which defaults to 16.

Leave catchup off when the Dag already works out its own backlog, for example a task that loads everything newer than the last row it saw. Turn it on when each run owns 1 interval of data and a missed interval is a hole in a table. Either way, the runs must be safe to repeat, which is the property described in idempotent pipelines.

A cron expression produces intervals aligned to the clock boundaries it names, whatever the start date. A timedelta does not. With catchup=False, a 30 minute delta aligns to the current time: if the Dag is enabled at 01:05, the first interval runs from 00:35 to 01:05, as the timetables documentation shows. The documentation also says DeltaDataIntervalTimetable doesn't necessarily align to arbitrary bounds such as the start of a day or an hour.

Pick a cron expression when the schedule has to match the clock, such as a report at 06:00, and a delta when only the spacing matters, such as polling every 10 minutes. If a Dag has to start the moment the last one ends, @continuous does exactly that.

When you move an existing Dag from a trigger timetable to a data interval timetable, the documentation says 1 scheduled run is skipped. The reverse change doesn't skip one.

Interviewers ask about the schedule because it exposes whether you know what a run represents. Common prompts: why does the run for yesterday start today, what's the data interval of a run you triggered by hand, and how would you reload 3 missed days without duplicating rows. The answers, in order: the interval ends before the run starts, a manual run has its own logical date, and each run must overwrite only its own interval.

Practice them with the Airflow interview questions and by designing the pipelines themselves in the data pipeline practice problems. The data pipeline interview questions cover the orchestration questions around them, and the system design round guide shows where scheduling comes up in the loop. For the structure of the Dag itself, see Airflow Dags explained.