Airflow DAG
Complete Reference for Data Engineers (2026)
A DAG, short for directed acyclic graph, is Airflow's unit of design: a Python file that defines your pipeline's tasks and the order they run in. Directed: every dependency points one way. Acyclic: no path ever loops back to an earlier task. Graph: tasks are the nodes, dependencies are the edges. This reference works through a complete example, scheduling semantics, dependency patterns, and debugging, plus the interview questions that test whether you have run DAGs in production.
The Anatomy of a DAG
Every box is a task and every arrow is a dependency you write with >>. Branches run in parallel, take uneven paths, and rejoin; the whole structure, not any single task, is what the scheduler executes.
extract fans out into two parallel branches that are deliberately uneven: the events branch takes an extra enrichment step before both rejoin at validate, and branches never need matching lengths. Downstream of the check the two siblings diverge by trigger rule: load keeps the default all_success, while cleanup runs under all_done so it fires even when validation fails. Every arrow still points forward; a cycle would make the run unschedulable, which is what acyclic forbids.
Know Airflow DAGs the way the interviewer who asks it knows it.
What Is an Airflow DAG?
An Airflow DAG is a Python file that defines a data pipeline as a directed acyclic graph: a set of tasks (the nodes) connected by dependencies (the edges). Directed means every edge has an order, so extract runs before transform. Acyclic means no loops, so a task can never depend on itself through any chain. The DAG itself does no work. It is a blueprint that describes what should happen and in what order; operators inside its tasks do the actual work.
The Airflow scheduler continuously imports every file in the dags_folder and rebuilds its picture of each DAG. On the DAG's schedule it creates a DagRun (one concrete execution, stamped with a logical_date), and each DagRun creates one TaskInstance per task. TaskInstances move through states (queued, running, success, failed, up_for_retry) as the executor runs them.
That split between definition and execution is the core mental model. The DAG file answers exactly two questions: what are the tasks, and what order do they run in. Everything operational (retries, backfills, schedules, alerting) hangs off that structure, which is why interviewers treat the DAG as a system-design artifact and not a syntax quiz.
Airflow DAG Example: A Complete Pipeline
from airflow.decorators import dag, task
from datetime import datetime, timedelta
@dag(
schedule="@daily",
start_date=datetime(2026, 1, 1),
catchup=False,
default_args={
"retries": 2,
"retry_delay": timedelta(minutes=5),
},
tags=["signups"],
)
def signups_daily():
@task
def extract(ds: str) -> str:
"""Pull one day of signups from the API into object storage."""
path = f"s3://lake/raw/signups/{ds}.parquet"
# fetch + write happen here; only the PATH crosses task boundaries
return path
@task
def transform(raw_path: str) -> str:
"""Dedupe on signup_id, normalize timestamps, write to staging."""
staged = raw_path.replace("/raw/", "/staged/")
return staged
@task
def load(staged_path: str) -> None:
"""MERGE into warehouse.signups keyed on signup_id. Safe to re-run."""
load(transform(extract()))
signups_daily()A production-shaped DAG in about 25 lines using the TaskFlow API. Dependencies are inferred from the function calls, XCom carries only the S3 paths, and the MERGE keeps the load idempotent so retries never duplicate rows.
Example DAG 1: A Daily ETL With a Quality Gate
The signups_daily pipeline drawn as a dataflow graph: one source, 3 tasks, a quality gate, and the storage each task reads and writes. This is the same canvas used in pipeline architecture interview rounds.
The signups_daily DAG from the code example, with a quality gate added between transform and load. Every annotation maps to a task argument: retries and retry_delay on extract, the partition overwrite and MERGE that keep re-runs idempotent, and the SLA the DagRun must hit.
How to Read a DAG on the Pipeline Canvas
The canvas vocabulary maps one-to-one onto Airflow. Rectangles are tasks: each becomes an operator or an @task callable, and its annotations (retries, backoff, idempotency strategy, SLA) become the task's arguments. Cylinders are systems outside Airflow: the API you pull from, the S3 staging area, the warehouse you load. Diamonds are quality gates, which in Airflow are just tasks whose job is to fail loudly, so the default all_success trigger rule halts everything downstream. Arrows are the dependencies you write with >>.
One mapping matters more than the rest: arrows order work, they do not move data. In the daily signups diagram, the extract task writes to S3 and the transform task reads from S3; the arrow between the tasks only guarantees the write finishes before the read starts. XCom carries the storage path across that edge, never the rows themselves. Interviewers probe this exact distinction because candidates who have only read tutorials assume the DAG edge is a data channel.
The three-source fan-in diagram is the shape behind most real warehouse loads and most pipeline design interview questions. Three extracts run in parallel because no edge connects them, which in code is one line: [extract_users, extract_events, extract_billing] >> build_marts, with each extract fed by a source that arrives on its own timetable. The dashed columns group tasks the way Airflow TaskGroups do, and the join task's all_success rule is what makes it wait for the slowest source.
Example DAG 2: Fan-Out, Fan-In Across Three Sources
Three sources that arrive at different times, three parallel extract tasks, and a join that fires only when every input has landed. The dashed columns play the role of Airflow TaskGroups.
The billing extract carries a reschedule-mode sensor because its source arrives late; its timeout alerts instead of hanging. The join is idempotent via DELETE + INSERT scoped to the data interval, so clearing any day re-runs cleanly.
How to Create an Airflow DAG, Step by Step
From an empty file to a monitored production run.
- 01
Define the DAG shell
Create a file in dags_folder and declare the @dag decorator (or a DAG context manager) with dag_id, schedule, start_date, and catchup=False. This is the contract the scheduler reads.
- ▸Always set catchup=False explicitly; an old start_date with catchup on floods missed runs the moment you unpause
- ▸Put retries and retry_delay in default_args so every task inherits them
- 02
Write the task callables
One @task function per unit of work: extract, transform, load. Keep each function idempotent and keep heavy imports (pandas, boto3) inside the function body so parsing stays fast.
- ▸Return paths and counts, not datasets; large data goes to object storage
- 03
Wire the dependencies
With TaskFlow, calling one task with another's return value creates the edge. With classic operators, use extract >> transform >> load, and lists for fan-out and fan-in.
- ▸Dependencies control order only; they never move data between tasks
- 04
Test before deploying
Run python dags/my_dag.py to catch import errors, airflow tasks test to execute a single task without the scheduler, and dag.test() for an end-to-end run in one process.
- ▸A parse-and-structure unit test in CI (task count + dependencies) catches most breakage
- 05
Deploy and watch the first run
Drop the file into the deployed dags_folder, wait for the scheduler to pick it up, unpause the DAG, and watch the first DagRun in the Grid view. Read the logs of every task once before trusting it.
- ▸If the DAG does not appear, run airflow dags list-import-errors before anything else
Airflow DAG Schedule Options
What each schedule value means and the trap that comes with it.
| Schedule | Meaning | Watch out for |
|---|---|---|
| @daily | Once per day at midnight UTC | The run for Monday fires Tuesday 00:00, at the end of Monday's data interval |
| 0 6 * * * (cron) | Exact cron control, here 06:00 UTC | Same interval-end semantics as presets; the 06:00 run covers the previous interval |
| timedelta(hours=4) | Fixed-length intervals measured from start_date | Not calendar-aligned; runs drift relative to clock-friendly times |
| None | Never scheduled; runs only when triggered manually or by another DAG | Easy to forget it needs a TriggerDagRunOperator, API call, or human |
| Datasets (Airflow 2.4+) | Event-driven: runs when upstream DAGs update the datasets it consumes | Producers must declare outlets, or the consuming DAG never fires |
logical_date, Data Intervals, and Catchup
The single biggest source of Airflow confusion is that a DagRun is stamped with the start of the interval it covers, not the moment it executes. A daily DAG's run for 2026-01-15 starts executing at 2026-01-16T00:00, once the day it is responsible for has fully elapsed. That stamp is the logical_date (called execution_date before Airflow 2.2), and templates like {{ ds }} render it, not today's date.
This design is what makes backfills coherent: re-running the 2026-01-15 DagRun months later still processes 2026-01-15's data, because every query is parameterized by the interval rather than the wall clock. Write your SQL against data_interval_start and data_interval_end and off-by-one-day bugs mostly disappear.
Catchup is the same machinery pointed backwards. When a DAG is unpaused, Airflow schedules a DagRun for every interval between start_date and now unless catchup=False. Keep catchup off by default and backfill on purpose with the airflow dags backfill command, which gives you date bounds and concurrency control.
Inside a DAG: The Building Blocks
The pieces every DAG is assembled from, and the one-line version of how not to misuse each.
Operators
An operator defines what one task does. PythonOperator runs a callable, BashOperator runs a shell command, SQLExecuteQueryOperator runs SQL, and provider packages add operators for AWS, GCP, Slack, and hundreds of other systems. Most pipelines need nothing exotic: PythonOperator (or the @task decorator that wraps it) covers the majority of real tasks.
Task Dependencies
Dependencies are the edges of the graph, written with the bitshift operator: extract >> transform >> load. Lists express fan-out and fan-in. Dependencies control order only. They do not pass data, so a downstream task never automatically receives its upstream's output.
Sensors
Sensors are operators that wait for a condition: a file landing in S3, a task in another DAG finishing, an API turning healthy. In poke mode a sensor holds a worker slot while it waits; in reschedule mode it releases the slot between checks. For any wait longer than a few minutes, use reschedule mode or you will starve the workers.
XComs
XComs move small values between tasks through the metadata database. With the TaskFlow API, return values are pushed and function arguments are pulled automatically. XComs are for metadata: paths, row counts, run ids. Passing a DataFrame through XCom is the classic anti-pattern; write the data to object storage and pass the path instead.
Trigger Rules
By default a task runs only when every upstream task succeeded (all_success). Other rules unlock error handling: all_done for cleanup that must always run, one_failed for alerting, none_failed to tolerate skipped branches. Trigger rules are how a DAG expresses what should happen when things go wrong.
Dynamic Task Mapping
expand() creates one task instance per element of a runtime value: one per file, one per table, one per partition. Unlike generating tasks in a loop at parse time, the mapped width can change on every run and the source list never blocks DAG parsing. Available since Airflow 2.3, and the right answer to almost every 'one task per X' problem.
Task Dependency Patterns
# Linear chain: strict order
extract >> transform >> load
# Fan out, then fan in: parallel transforms behind one extract
extract >> [clean_users, clean_events] >> build_marts
# Cleanup that runs whatever happened upstream
cleanup = PythonOperator(
task_id="cleanup",
python_callable=drop_staging,
trigger_rule="all_done",
)
[clean_users, clean_events, build_marts] >> cleanup
# Branching: choose a path at runtime
def choose(**context):
return "full_refresh" if context["params"].get("full") else "incremental"
branch = BranchPythonOperator(task_id="choose_path", python_callable=choose)
branch >> [full_refresh, incremental]
# Dynamic task mapping (Airflow 2.3+): one task instance per input,
# with the input list decided at runtime, not parse time
@task
def list_files() -> list[str]:
return ["a.csv", "b.csv", "c.csv"]
@task
def process(path: str) -> None:
...
process.expand(path=list_files())The five dependency shapes that cover nearly every real DAG: chains, fan-out/fan-in, always-run cleanup, runtime branching, and dynamic task mapping. Shown side by side for reference, not as one runnable file.
Debugging a Misbehaving DAG
The failure modes that account for most Airflow support threads.
| Symptom | Likely cause | Fix |
|---|---|---|
| DAG never appears in the UI | Import error, or the file is outside dags_folder | Run airflow dags list-import-errors; confirm the file defines a DAG object at module level |
| Tasks sit in queued forever | Pool slots, max_active_tasks, or parallelism exhausted (or the executor is down) | Check pool usage in the UI, raise the relevant limit, and confirm workers are heartbeating |
| Run processes the wrong day's data | Templates read logical_date as 'today' | Parameterize queries with data_interval_start / data_interval_end, not the wall clock |
| Unpausing floods hundreds of runs | catchup=True with an old start_date backfills every missed interval | Set catchup=False and backfill deliberately with airflow dags backfill |
| Whole DAG stalls on one sensor | Poke-mode sensor holding a worker slot for hours | Switch to mode='reschedule' with a sane poke_interval and timeout |
| Scheduler is slow, every DAG starts late | Top-level code in DAG files: API calls, DB reads, heavy imports at parse time | Move work inside task callables and lazy-import heavy libraries inside functions |
Airflow DAG Best Practices
The rules that separate a demo DAG from one that survives production.
- Make every task idempotent. Airflow retries failed tasks and re-runs cleared ones, so a task that INSERTs blindly creates duplicates. Use MERGE or upsert, DELETE + INSERT in one transaction, or overwrite a date-partitioned path. Idempotent tasks are safe to retry, safe to backfill, and safe to clear while debugging.
- Configure retries with backoff. Transient failures (network timeouts, rate limits, lock contention) are normal. retries=2 with retry_delay=timedelta(minutes=5) absorbs most of them without a human. For flaky external APIs, add retry_exponential_backoff=True with a max_retry_delay cap.
- Keep the top level of the file inert. The scheduler re-imports every DAG file continuously, so top-level code runs constantly. The module level should only build the DAG object: no API calls, no database reads, no heavy imports. All real work belongs inside task callables.
- Write new DAGs with the TaskFlow API. The @task decorator infers dependencies from function calls and handles XCom automatically, so the pipeline reads like plain Python. Fall back to classic operators for sensors and provider operators; mixing both styles in one DAG works fine.
- Guard shared systems with pools. If the warehouse tolerates 10 concurrent connections, create a 10-slot pool and assign every warehouse-writing task to it, across all DAGs. Pools are the global brake that stops 50 simultaneous tasks from crashing a shared resource.
- Split DAGs that grow past ~40 tasks. Huge DAGs parse slowly, clutter the UI, and are painful to debug. Break them into focused DAGs connected by Datasets or TriggerDagRunOperator, each representing one coherent unit of work.
DAG questions rarely stay on the DAG. An interviewer who asks what a DAG is will follow with how you would backfill it, what happens when a task is retried after a partial write, and how the schedule interacts with late-arriving data. Those follow-ups are the actual data pipeline interview questions an orchestration round is built from, and they show up in roughly half of data engineer loops. If you are preparing for a specific onsite, the round-by-round interview prep guide covers where orchestration sits in the wider loop.
Reading about idempotency is not the same as having written a task that survives its second run. Working through the data pipeline practice problems forces the retry and backfill cases into code, which is where the reasoning above stops being abstract.
Airflow DAG Interview Questions
Questions that check your grasp of DAGs beyond the basics.
Explain the difference between logical_date (execution_date) and the actual time a DAG runs. Why does this distinction matter?
The logical_date marks the start of the data interval, not when the DAG runs. A daily DAG with logical_date 2026-01-15 runs on 2026-01-16 at midnight, at the end of its interval. This matters because SQL parameterized with the logical_date processes the correct day's data; misreading it is a common source of off-by-one-day bugs. Use data_interval_start and data_interval_end in templates for clarity. The interviewer is looking for someone who has hit this in production.
Your DAG runs daily, but yesterday's run failed and today's run is queued. Walk through what happens and how you fix it.
By default (depends_on_past=False), today's run executes regardless of yesterday's failure. If depends_on_past=True on any task, that task waits for its previous DagRun to succeed. To fix the failure: read the failed task's logs, fix the root cause, then clear the failed task instances, which re-queues them. The interviewer is checking whether you know that clearing a task re-runs it while marking it success skips it, and whether you would clear just the failed task or the whole DagRun.
How would you design a DAG that handles failures gracefully, including sending an alert and running a cleanup task?
Layered handling. Retries with retry_delay on each task absorb transient failures. on_failure_callback at the DAG level sends the alert (Slack, PagerDuty, email) once retries are exhausted. A cleanup task with trigger_rule='all_done' runs regardless of upstream state. For critical outputs, a downstream check verifies the expected data actually exists. The interviewer wants retries for the transient, alerts for the persistent, cleanup for every outcome.
You need to process data from 3 sources, each arriving at different times, then join them. How do you design this DAG?
Three extraction tasks in parallel, each writing to a staging location. The join task depends on all three with the default all_success rule, reads from staging, and produces the final output. Wrap each source in a reschedule-mode sensor with a timeout and an alerting callback so a late source pages someone instead of hanging silently. The interviewer is checking whether you account for the messy cases: sources that arrive late, with drifted schemas, or with overlapping data.
Airflow DAG FAQ
What is the difference between a DAG and a pipeline?+
What does catchup=False actually do?+
How many tasks should a single DAG have?+
Should I use the TaskFlow API or the classic operator style?+
How do I trigger a DAG manually?+
How do I test Airflow DAGs?+
Practice Pipeline Architecture Questions
- 01
Reading a solution is not the same as writing one
Every engineer who has frozen on a query they had read a dozen times knows the gap. The only preparation that closes it is producing the answer yourself, under time, before the interview does it for you
- 02
76% of hiring managers reject on the coding task, not the resume
From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice
- 03
System design comes down to the calls you defend out loud
Ingestion, batch vs streaming, the bronze/silver/gold layers, idempotency, backfill and replay. Sketching the pipeline and naming the failure modes is the signal, not the boxes
Related Guides
Scheduler, executors, XComs, and production reliability questions
End-to-end pipeline design patterns for data engineering
Pipeline architecture, batch vs streaming, and scale reasoning
1 data engineering challenge a week on dirty production-shaped data, scored blind on a hidden dataset. Submit before the freeze; results at the reveal.