Airflow Tutorial

A DAG file has 3 parts: the imports, the DAG object that carries the schedule, and the tasks with the dependencies between them. The dag_id is the name you see in the interface. schedule says when runs are created. start_date is the earliest point a schedule can use, and catchup=False stops Airflow from creating a run for every missed interval since that date.

Each task is an instance of an operator, a predefined template for one kind of work. The BashOperator runs a shell command, and its command string is a Jinja template, so {{ ds }} renders as the run's logical date in YYYY-MM-DD form. The >> operator sets order: extract >> transform >> load means transform starts only after extract succeeds. With no other setting, a task runs when all of its upstream tasks have succeeded.

The fan-out form works too. The Airflow fundamentals tutorial writes t1 >> [t2, t3] to run t2 and t3 in parallel after t1. Save the file in the dags folder and run it once with python to confirm it parses: an import error or a syntax error shows up immediately, before the scheduler ever sees the file.

The TaskFlow API turns a Python function into a task with the @task decorator, and a @dag function into the DAG. You import dag and task from airflow.sdk. Dependencies come from the calls themselves: writing transform(extract()) makes transform wait for extract, so most TaskFlow DAGs need no >> at all.

The values pass between tasks too. The documentation states that a task's return value is stored as an XCom and handed to the downstream task automatically. With multiple_outputs=True, a returned dictionary is split into one XCom per key, so a later task can take result["total_order_value"] alone.

Choose by the work. A task that runs a shell command or calls a provider's operator is clearest as an operator. A task that is a few lines of Python is clearest as a decorated function. Both styles live in one DAG, and TaskFlow tasks can depend on operator tasks.

When a DAG is first switched on, the scheduler creates a run only for the latest interval, because the scheduler.catchup_by_default setting is False. Set catchup=True on the DAG and the scheduler creates a run for every interval since the last one that ran, which also happens when you pause a DAG for a period and switch it back on.

A backfill is the deliberate version: you choose a historical period and Airflow re-runs the DAG over it. The command is airflow backfill create, which takes a from date and a to date. Further options set reprocess behavior and maximum active runs, or run the period backwards. The Trigger menu on the DAG details page opens a backfill form in the interface too.

Both features run your code many times over old intervals, so they only work safely when a task can run twice with the same outcome. The idempotency guide covers how to design for that.

The DAG processor parses each file in the dags folder and writes the result to the metadata database. The scheduler reads those records, creates DAG runs on schedule and submits ready tasks to the executor. The executor is a configuration property of the scheduler and runs inside the scheduler process, so it's a setting, not a separate service.

A task instance moves through states as this happens. The documentation lists 14, among them scheduled and queued, then running, success, failed, up_for_retry and skipped, and finally upstream_failed. The last one tells you a task never ran because something it depended on failed, which is where to look for the real error.

Because the dag processor re-parses files repeatedly, a DAG file is not the place for heavy work. The documentation says not to run heavy computation or database and network calls at the top level of a DAG file, since that code runs on every parse and slows the scheduler. Put the work inside tasks.

Tasks are isolated, each in its own process and often on its own machine. An XCom, short for cross-communication, is the mechanism that lets them talk: a value stored against a key, scoped by task ID and DAG ID, readable by later tasks in the same run. TaskFlow return values use it implicitly. Traditional operators call xcom_pull() explicitly.

The documentation says XComs are designed for small amounts of data and warns against passing large values such as dataframes. Pass a reference instead: write the large result to storage and return its path or table name, so the next task reads from there. For larger payloads, the documentation points to the object storage XCom backend, and the backend is swappable through the xcom_backend configuration option.

Retries are set per task or in the DAG's default_args. The same tutorial sets retries to 1 with a retry_delay of 5 minutes, and overrides it to 3 retries on one task. A task waiting for its next attempt shows the up_for_retry state.

Retries make one rule non-negotiable. The documentation puts it directly: Airflow can retry a task if it fails, so the task should produce the same outcome on every re-run. It recommends UPSERT in place of INSERT, since an insert can duplicate rows, and reading the partition for data_interval_start instead of the latest available data. A task written this way survives retries, and the re-runs that catchup and backfill cause, without cleanup. The idempotency guide works through the patterns.

Test in 3 widening steps. First run the file with python so imports and syntax fail loudly. Then check what Airflow sees with airflow dags list and airflow tasks list. Finally exercise real work: airflow tasks test runs one task instance for a logical date, and airflow dags test runs a whole DAG run locally and records its state in the metadata database.

The documentation describes dag.test() as an integration test that needs a metadata database, so it complements unit tests of your task functions and does not replace them. Keep the logic of a TaskFlow task in plain functions you can call from a test with ordinary arguments.

Interviewers use Airflow to probe whether you can reason about pipeline behavior: what a retry does to a non-idempotent task, what a backfill touches, why a daily run starts after midnight. The data pipeline interview questions and data pipeline practice problems put those ideas under time pressure, and the Airflow interview questions go deep on the tool itself.

The Airflow DAG guide covers the DAG object and its runs in more detail. When a design question asks how you would operate a pipeline, the system design round is the format to rehearse.