A Run Covers a Slice of Time

Concepts covered: Data intervals

One run, one window

A scheduled run of a DAG answers one question: what happened in this slice of time? The slice has a start and an end, and Airflow calls it the run's data interval. A DAG that runs once an hour owns 1 hour per run, and the tasks inside that run read the rows that landed within that hour and nothing outside it. 3 time values ride along with every run: data_interval_start, data_interval_end and logical_date. For every scheduled run logical_date equals data_interval_start, so the run is named after the moment its window opened; runs triggered by hand, covered in the last section, follow a different rule.

Airflow creates the run when the window closes. The run that owns 09:00 to 10:00 on 4 November 2026 is created at 10:00, because by then every click from that hour has been written and the hour can be counted whole. A run created at 09:00 for the same hour would count an hour that had hardly begun, and the figure it produced would be stale by the time anyone read it.

Read down the table and the pattern holds in every row: the name inside the run_id is the time under Window opens. Window closes and Run is created always hold the same moment seen from 2 sides, the end of one window and the first moment the run for it can exist. Whenever this lesson asks which runs exist at a given moment, the answer comes from reading a table like this one.

import pendulum
from datetime import timedelta
from airflow.sdk import dag, task

@dag(
    dag_id="hourly_clicks",
    start_date=pendulum.datetime(
        2026, 11, 4, tz="UTC"),
    schedule=timedelta(hours=1),
)
def hourly_clicks():
    @task
    def extract():
        ...
    @task
    def load():
        ...
    extract() >> load()

hourly_clicks()

Watch where each cell appears in the animation above: the run named 00:00 lights up at 01:00, the run named 01:00 lights up at 02:00, and so on down the grid. The file names a start_date of midnight and a schedule of 1 hour, and Airflow lays the hours end to end from there. Nothing in the file states when a run starts; that follows from where its window ends.

This gives you a way to read any moment on the clock. At 11:40 on 4 November, the runs that exist are the ones whose windows closed at or before 11:40: the runs named 10:00, 09:00 and every hour back to start_date. The 11:00 window is still open, so there is no run named 11:00 yet, and there will be none until 12:00. A grid that ends with the 10:00 column at 11:40 is a grid that is exactly up to date.

Every time in this lesson is UTC, which is what tz="UTC" on each start_date declares. Other time zones change where the boundaries fall and are covered in a later lesson.

Reading the window inside a task

A task gets its run's window by asking for it. With TaskFlow, name a parameter data_interval_start or data_interval_end and Airflow fills it with the run's value when the task starts, so the same function reads a different hour in every run with no change to the file. Compare that with datetime.now() called inside the task, which returns the moment the task happened to execute, 10:03 in one run and 10:07 in a slower one, and returns something else again if the task is run a second time.

@task
def count_clicks(data_interval_start=None, data_interval_end=None):
    print(f"counting {data_interval_start} to {data_interval_end}")

Naming the run by its window is what makes a second run of it meaningful. Run scheduled__2026-11-04T09:00:00+00:00 again a week later and it is handed the same 2 values, counts the same hour and writes the same answer. The clock time it ran at has moved by 7 days and nothing it reads has moved at all. A task that trusted the clock instead would count a different hour on every attempt.

About this lesson

This section is part of the Scheduling: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.