Why a DAG Goes Missing

Concepts covered: DAG parsing

The file is in the folder, the processor has swept it a dozen times, and the home page shows no DAG with its id. Every cause leaves a different fingerprint, and a short, fixed order of checks finds it in a few minutes. The order matters because the first checks are cheap and rule out the common causes, and the later checks only make sense once the earlier ones have come back clean.

Cause 1: the import raised

An exception at module level stops the import, and the processor records it against the file path in the import error table. The UI shows an import error notice on the home page, and airflow dags list-import-errors prints the file path with the full traceback. Missing modules and mistyped names land here, as does a KeyError on an environment variable the processor's host lacks. A file that errors produces no DAG, so a new DAG from a broken file is absent from the list rather than shown in red.

from airflow.sdk import DAG, tsk

with DAG(dag_id="orders_health", schedule=None) as dag:
    @tsk
    def check_orders():
        print("ok")

    check_orders()

Cause 2: the file was never imported

The processor can skip a file without ever opening it, and it records no error when it does. Before importing, it scans the file's text for the strings airflow and dag, case-insensitive, and skips any file lacking either one. A file that gets its DAG by calling a factory function from another module can fail this test and sit there in silence. A .airflowignore file in the folder or any parent folder also holds glob patterns, and a file matching one is skipped, along with everything under a matching directory. And the folder the processor scans is whatever airflow config get-value core dags_folder prints, so a file 1 directory outside it does not exist to Airflow.

from pipelines.factory import build_orders_health

orders_health = build_orders_health()

Cause 3: the DAG object never reached the module

The processor collects DAG objects from the module's global namespace after the import. A @dag decorated function that is defined and never called creates no DAG, because the decorator turns the function into a factory, and the DAG exists only when the factory runs. A with DAG(...) block inside a function the module never calls has the same effect. In both cases the import succeeds, there is no error to list, and the file parses in milliseconds, so this cause hides behind a clean report.

from airflow.sdk import dag, task

@dag(dag_id="orders_health", schedule=None)
def orders_health():
    @task
    def check_orders():
        print("ok")

    check_orders()

Cause 4: another file owns the id

A dag_id is unique across the whole database. If 2 files define the same id, both import cleanly. The DAG then shows under whichever file the processor parsed last, with that file's tasks. The other file's tasks never appear, and the DAG's recorded file path can change from one sweep to the next. Copying a DAG file to start a new one and forgetting to change the id produces this exactly, and airflow dags list shows the id once, with a file path you did not expect.

Cause 5: the import ran out of time

An import that passes 50 seconds is killed and the DAG is never written. The dag processor's log records it with a line like Processor for dags/bucket_inventory.py with PID 4182 started 51 ago killing it., which names the file path and its age in seconds, past the 50-second limit. Causes 2 and 3 leave no line in that log at all, which is itself a clue. The usual source of a timeout is the module-level work from the previous section: a slow query, a listing of a large bucket, an import chain that pulls in a numerical library. The fix is the same as before, moving the work into a task, and the check is airflow dags report, which prints the file's parse duration next to its name.

A DAG that appears with its toggle off has arrived and is paused. New DAGs start paused by default, which is a different situation from a DAG that never arrived.

After fixing the file, you can wait for the next sweep or force one. airflow dags reserialize reparses every file in the folder and rewrites the serialized DAGs, which is the quickest way to confirm a fix on a deployment where sweeps run slow. If the DAG still fails to appear after a forced reserialize, the cause is one of the 5 above, and the order of checks will find it.

About this lesson

This section is part of the DAG Anatomy: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.