Airflow DAG

The @dag decorator turns the function signups_daily into a DAG whose id is the function's name. Calling the function at the bottom of the file creates the DAG object at module level, which is the only place Airflow looks for one. The schedule argument uses CronDataIntervalTimetable on purpose: on Airflow 3 a bare "@daily" gives each run an interval of 0 length, and this DAG needs every run to cover exactly 1 finished day.

default_args apply to every task: 2 retries with 5 minutes between them, so a failing task gets 3 attempts and the last one starts at least 10 minutes after the first failure. Without that line each task gets 0 retries, the default. catchup=False stops the scheduler from creating a run for every day since start_date when the DAG is first unpaused.

Inside, each @task function becomes 1 task. extract asks for data_interval_start by naming it as a parameter, and Airflow fills it in from the run's context, so a rerun of any past day reads that day and not today. The expression load(transform(extract())) does 2 things at once: it creates the edges extract >> transform >> load, and it passes each return value to the next task as an XCom. Those values are paths, never the rows themselves.

The canvas maps onto Airflow 1 to 1. Transform nodes are tasks: each becomes an operator or a @task function, and its annotations (retries, backoff, idempotency, freshness) become that task's arguments. Source and storage nodes are systems outside Airflow: the API you pull from, the S3 staging area, the warehouse table you load. A quality gate is a task whose job is to fail loudly, and the default all_success rule then stops everything downstream of it.

1 mapping matters more than the rest: an arrow between 2 tasks orders the work, it does not move the data. In the daily signups DAG, extract_signups writes to S3 and transform_signups reads from S3; the edge only guarantees the write finishes before the read starts, and XCom carries the storage path across it. Candidates who have only read tutorials tend to treat the edge as a data channel, which is why interviewers ask.

The fan-in shape of the 3-source DAG is behind most warehouse loads and most data pipeline architecture patterns an interview asks you to draw. The 3 extracts run in parallel because no edge connects them, which in code is 1 line: [extract_users, extract_events, extract_billing] >> build_marts. The layers group tasks the way an Airflow TaskGroup does, and the join's all_success rule is what makes it wait for the slowest source.

In everyday use the 3 words overlap. A pipeline is the path data takes from source to destination, a workflow is any ordered set of steps, and a DAG is Airflow's definition of 1 workflow: its tasks, the order they may run in, and the settings that say when and how often. When an engineer says the DAG failed, they mean 1 run of that definition failed.

Airflow also schedules and pauses whole DAGs, and reports on each one as a unit. Retries and alerts attach to the DAG or its tasks, as do concurrency limits, and the UI groups every run and log under the DAG's id. The design decision is whether to split 1 pipeline into several DAGs or fold several into 1, because that decides what you can pause or rerun on its own and which failures alert separately.

A DAG says nothing about how data is transformed, only when each step runs. That's why teams often run SQL transformation tools inside an Airflow task rather than instead of Airflow, a split explained in dbt vs Airflow.

Airflow 3 splits the work between separate processes. The DAG processor imports every file in the DAGs folder and serialises the DAGs it finds, re-parsing each file about every 30 seconds by default. The scheduler reads those serialised DAGs, and when a schedule comes due it creates a DAG run: 1 concrete execution of the DAG, identified by its logical date and run id.

Each DAG run creates 1 task instance per task. A task instance moves through states as the scheduler and executor handle it, from scheduled to queued to running, and it finishes as success or failed unless it is skipped or goes to up for retry. The scheduler queues a task instance only when its trigger rule is satisfied by the states of its upstream task instances in the same run, which is how the edges of the graph become an order of execution.

Limits keep 1 DAG from taking over the cluster. By default a DAG runs at most 16 task instances at once (max_active_tasks) and keeps at most 16 DAG runs active (max_active_runs), and each scheduler runs at most 32 task instances in total (parallelism). Pools add a shared limit across DAGs: a 10 slot pool on the warehouse caps writes to it at 10, whichever DAGs they come from.

Every DAG run carries a logical_date, the timestamp {{ ds }} renders, and it's often not the moment the run starts. Under the classic data interval model, a daily run covers 1 whole day and starts only after that day ends: the run that starts at 00:00 on day 2 has a data interval from day 1 00:00 to day 2 00:00, and its logical date is the start of that interval, day 1, so the run labelled Monday starts on Tuesday.

Airflow 3 changed the default. The Airflow 3 upgrade guide sets create_cron_data_intervals to False, so a bare cron string or preset such as "@daily" now resolves to CronTriggerTimetable. The run still starts at 00:00 on day 2, but its logical date is day 2 00:00, and data_interval_start equals data_interval_end. Code that reads ds and expects yesterday now reads today, a day that has only just begun.

The fix is to say which model you mean. Pass CronDataIntervalTimetable("@daily", timezone="UTC") to keep the classic interval, or CronTriggerTimetable with an interval argument, whose data interval spans that duration and ends at the trigger time. Then parameterise every query with data_interval_start and data_interval_end rather than ds or the wall clock, and a rerun of any past run reads exactly the data it read the first time. execution_date, the old name for the logical date, is gone from the task context in Airflow 3.

On Airflow 3 the command airflow backfill create takes a DAG id and a date range and creates 1 run per interval in it, with options to rerun only failed runs, cap how many run at once and go newest first. Each backfilled run sees its own data interval, which is what makes a backfill correct, provided the tasks read that interval and not the current date.

Backfills and retries are only safe when rerunning a task for the same interval produces the same result. A task that appends rows doubles them on the second run; one that overwrites its partition or merges on a key does not. Interviewers treat that property as the dividing line between a demo DAG and a production one.

Tasks in a DAG are isolated: they may run on different machines, in different processes, minutes apart. XCom is how they hand each other small values. With the TaskFlow API a return value is pushed as an XCom automatically, and passing it as an argument to another task pulls it and creates the dependency in the same step.

By default XComs live in Airflow's metadata database, and the documentation is explicit that they are designed for small amounts of data, never large values such as dataframes. Write the data to object storage or a table, and pass along the path or the partition. A row count is small enough to pass too. A custom XCom backend can store larger values in object storage, but the path pattern keeps each task's output inspectable and each task rerunnable on its own.

A DAG that handles failure well layers 3 mechanisms. Retries with a delay absorb transient failures such as timeouts and rate limits. A failure callback on the DAG or its tasks alerts a person once retries are used up. A cleanup task with all_done drops temporary tables or files whatever happened. For outputs that matter, a final check task that verifies the data exists catches the runs that succeeded without writing anything.

A branch chooses a path at runtime. A function decorated with @task.branch returns the id of the task to follow and Airflow skips the others; it can also return a list of ids. Because skipped tasks are not successes, a task that joins the branches needs a trigger rule such as none_failed_min_one_success, or it will be skipped too.

Dynamic task mapping creates copies of a task at runtime. process.expand(path=list_files()) makes 1 task instance per element that list_files returns, whether that is 3 files today or 300 tomorrow, and each copy retries and logs on its own. It arrived in Airflow 2.3, and by default 1 expansion may create up to 1024 task instances. A for loop in the DAG file can also create tasks, but it runs at parse time, so the count is fixed when the file is read and any call it makes to list the inputs slows every parse.

Airflow reruns tasks constantly: retries after a failure, a cleared task during debugging, a backfill over last month. Each rerun is safe only when the task is idempotent, producing the same result for the same interval however many times it runs. The Airflow best practices guide puts it directly: an INSERT on a rerun can duplicate rows, so use an upsert, and never read the latest available data, because it may change between runs.

In practice that means 3 patterns: MERGE or upsert on a business key, delete then insert for the interval inside 1 transaction, or overwrite the partition the interval owns. Each ties the task's output to its data interval, never to the time it ran. The patterns and their trade-offs are covered in idempotent data pipelines.

Configure retries once, in default_args, since the default is 0. A few retries with a delay of a few minutes absorb most transient failures, and retry_exponential_backoff=True with a max_retry_delay cap suits flaky external APIs. Retries are only safe on idempotent tasks, which is why the 2 practices belong together.

The DAG processor re-imports every DAG file on a loop, about every 30 seconds by default, so any code at the top level of the file runs thousands of times a day. The top level should only build the DAG, so keep API calls, database queries, reads of Airflow Variables with Variable.get() and heavy imports such as pandas out of it. Put the work in task functions and reference Variables through templates such as {{ var.value.name }}, which resolve only when the task runs.

A slow file delays more than itself: the processor has less time for every other DAG, and changes take longer to show up. Keep 1 coherent unit of work per DAG, and when a DAG grows into unrelated parts, split it into separate DAGs connected by assets, so the downstream DAG runs when the upstream one updates its data.

Test at 3 levels. First, load the DAG in a unit test and assert that it imports without errors and has the task ids and edges you expect; running python on the file is the quickest version of the same check. This catches most breakage and belongs in CI.

Second, run 1 task in isolation with airflow tasks test, which executes it for a given logical date without the scheduler and without recording state. Third, run the whole DAG in 1 process with dag.test(), which runs every task in order and makes a debugger usable. Before a DAG goes to production, run it once in a staging environment against realistic data, including a rerun of the same interval to prove it's idempotent.

Airflow 3.0 shipped in April 2025, and most tutorials still describe Airflow 2. When you write or read a DAG, the imports change first: dag now comes from airflow.sdk, along with task and DAG. catchup defaults to False, and bare cron schedules resolve to CronTriggerTimetable, so the logical date is the moment a run starts. The context keys execution_date and prev_ds are gone, and so is next_ds.

Several features were renamed or removed. Datasets became Assets. SubDAGs are gone, replaced by TaskGroups. SLAs were removed in favour of Deadline Alerts. Task code can no longer query the metadata database directly and uses the Airflow Python client or the REST API instead. The DAG processor runs as its own process rather than inside the scheduler.

When you read an older DAG, those differences tell you its age at a glance: schedule_interval or execution_date dates it to Airflow 2 or earlier, and so do from airflow import DAG and a SubDagOperator.

DAG questions rarely stay on the definition. An interviewer who asks what a DAG is will follow with what happens when yesterday's run failed and today's is queued, how a DAG should alert and clean up after a failure, and how to join 3 sources that land at different times. The answers are the mechanisms on this page: independent runs unless depends_on_past is set, clearing a failed task instead of marking it success, retries plus a failure callback plus an all_done cleanup, and parallel extracts behind reschedule-mode sensors with timeouts feeding an idempotent join.

The strongest answers name the data interval. Saying that each run reads exactly its own interval, and that every write is a merge or a partition overwrite, shows you have operated DAGs rather than only written them. Work through the Airflow interview questions for the scheduler and executor follow-ups, and the data pipeline interview questions for the design questions a DAG answers.

Designing the DAG out loud is its own skill. The data pipeline practice problems ask you to draw pipelines like the 2 example DAGs on the same canvas, and the system design round guide covers where orchestration questions sit in the loop.