DAG Versions and Bundles

Concepts covered: DAG parsing

A bundle names where the files come from

Since Airflow 3, the DAG processor doesn't watch a bare folder. It watches bundles, each a named source of DAG files with a class that knows how to fetch them. A fresh install has 1 bundle, dags-folder, of class LocalDagBundle, pointing at the dags folder on disk, which is why every earlier lesson worked without mentioning one. The list lives in the [dag_processor] section of the configuration as dag_bundle_config_list, and airflow dags list prints a bundle_name column so you can see which bundle each DAG came from.

[
  {
    "name": "dags-folder",
    "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
    "kwargs": {}
  },
  {
    "name": "pipelines",
    "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
    "kwargs": {
      "git_conn_id": "pipelines_repo",
      "tracking_ref": "main",
      "subdir": "dags",
      "refresh_interval": 60
    }
  }
]

A GitDagBundle, shipped in the git provider, tracks a branch of a repository. Its git_conn_id names the Airflow connection that holds the repository's URL and credentials, and without it the bundle has nothing to clone. Its version is the commit hash it last fetched, and it checks for a newer commit every refresh_interval seconds, 60 here. A LocalDagBundle has no version, because a folder on disk holds no record of which copy of a file it contained at 02:00. That one difference decides everything else in this section.

A version per change

Every time the processor serializes a DAG and finds the result differs from the stored one, it records a new DAG version with a rising version number, and the same happens when the bundle's version changes under an unchanged file. The UI's Code tab lets you pick any version and read the file as it was then. Each run records the version it was created against, and each task instance records the version it executed under, which is what lets the grid show a run and say what code it was meant to execute. Any change to the tasks or the arrows between them, a renamed task id included, changes the serialized form; editing the body of a function inside a @task does not, because the serialized DAG holds the function's name and not its code.

The run that straddles a deploy

A run of load_orders, whose 3 tasks are now extract, load and verify, starts at 02:00 against version 3. At 02:10 extract has finished and load is running when a deploy lands version 4, which fixes a bug in verify's body and adds an arrow from extract straight to verify. The question is which verify runs at 02:12, and the answer depends on the bundle.

With the local bundle, the worker opens whatever file is on disk when verify starts, so it runs the fixed body even though the run was created against version 3. The run has mixed code: extract and load from version 3, verify from version 4, and the task instance detail shows the version each one executed under. With the Git bundle, the run recorded commit c41, the worker fetches that commit before starting verify, and the run finishes on version 3 code throughout, bug included. The fix reaches the 02:30 run, which is created against version 4. Neither behavior is wrong; one gives you the fix 18 minutes sooner, the other gives you a run whose code you can name with 1 hash.

Reading this back from the UI takes 2 clicks. The run's detail names the version it was created against; each task instance names the version it ran under; and a run showing 2 different numbers is a run that crossed a deploy. The Code tab, set to the older number, shows the file that run was planned from, which is the file to read when a task that passed yesterday fails tonight.

About this lesson

This section is part of the DAG Anatomy: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.