Airflow DAG: Complete Reference for Data Engineers (2026)

Operators define what a task does. PythonOperator runs a Python callable, BashOperator runs a shell command, SqlOperator runs a SQL query. There are also transfer operators (move data between systems), sensor operators (wait for conditions), and provider operators (AWS, GCP, Slack). Most tasks don't need a custom operator; PythonOperator covers the majority of cases. PythonOperator(task_id='transform', python_callable=transform_data, op_kwargs={'date': '{{ ds }}'})

A task is an instance of an operator within a DAG. Tasks are connected by dependencies that define execution order. Use >> (bitshift operator) to set dependencies: extract >> transform >> load means extract runs first, then transform, then load. You can also fan out (extract >> [transform_a, transform_b]) or fan in ([transform_a, transform_b] >> load). Dependencies only control order, not data passing. Tasks do not automatically receive output from upstream tasks. extract >> [transform_users, transform_events] >> load_warehouse

Sensors are special operators that wait for an external condition to be true. FileSensor waits for a file to appear. ExternalTaskSensor waits for a task in another DAG to complete. HttpSensor waits for an HTTP endpoint to return a success status. Sensors have two modes: poke (checks repeatedly, holding a worker slot) and reschedule (releases the worker between checks). Use reschedule mode for long waits to avoid blocking worker slots. A sensor that pokes every 30 seconds for 6 hours ties up a worker slot the entire time. FileSensor(task_id='wait_for_file', filepath='/data/input/{{ ds }}.csv', mode='reschedule', poke_interval=300)

XComs let tasks pass small pieces of data to downstream tasks via the Airflow metastore (database). A task pushes a value with xcom_push, and a downstream task pulls it with xcom_pull. The TaskFlow API (@task decorator) handles XCom automatically: the return value of a decorated function is pushed, and calling the function in another task pulls it. XComs are not for large data. They store values in the metastore database. Passing a DataFrame through XCom is an anti-pattern. Instead, write data to object storage (S3, GCS) and pass the file path as an XCom. ti.xcom_push(key='row_count', value=42) / ti.xcom_pull(task_ids='extract', key='row_count')

The default trigger rule is all_success: a task runs only when all upstream tasks succeed. This is correct for most cases, but sometimes you need different behavior. all_failed: run only if all upstreams failed (useful for error notification tasks). one_success: run if any upstream succeeded (useful for branching). none_failed: run if no upstream failed (allows skipped upstreams). always: run regardless of upstream state (useful for cleanup tasks). Understanding trigger rules is essential for building DAGs with error handling and branching logic. PythonOperator(task_id='cleanup', python_callable=cleanup, trigger_rule='all_done')

Every task should produce the same result regardless of how many times it runs. This is critical because Airflow retries failed tasks and operators can manually re-run tasks. If your task inserts rows into a database, a retry creates duplicates. Instead, use MERGE/upsert, or DELETE + INSERT within a transaction, or write to a date-partitioned path and overwrite the partition. Idempotent tasks are safe to retry, safe to backfill, and safe to re-run when debugging.

Every production task should have retries configured. Transient failures (network timeouts, API rate limits, temporary database locks) are normal. A task with retries=3 and retry_delay=timedelta(minutes=5) handles most transient failures automatically without human intervention. For tasks that call external APIs, use exponential backoff: retry_delay doubles with each attempt. Set retry_exponential_backoff=True and max_retry_delay to cap it.

The scheduler parses DAG files every few seconds. If your DAG file makes API calls, reads from a database, or imports heavy libraries at the top level, you slow down the scheduler for every DAG. Move all work into task callables. Use lazy imports (import inside the function, not at the top of the file) for heavy libraries like pandas or boto3. A DAG file should define tasks and dependencies and nothing else.

The TaskFlow API (@task decorator, introduced in Airflow 2.0) simplifies DAG authoring. Instead of instantiating PythonOperator and passing callables, decorate Python functions with @task. Return values become XComs and dependencies get inferred from function calls. The code reads more naturally. Use TaskFlow for new DAGs unless there's a specific reason to use the classic operator style.

Pools limit concurrent task instances across all DAGs. If your data warehouse allows 10 concurrent connections, create a pool with 10 slots and assign all warehouse-writing tasks to that pool. This prevents 50 tasks from trying to write simultaneously and crashing the database. Pools are global: they apply across all DAGs and all DagRuns. Use priority_weight to control which tasks get pool slots first.

A DAG with 200 tasks and complex branching logic is hard to debug, slow to parse, and difficult for anyone else on the team to understand. If your DAG is growing beyond 30 to 40 tasks, consider breaking it into multiple DAGs connected by Datasets or TriggerDagRunOperator. Smaller, focused DAGs are easier to test, faster to troubleshoot, and more maintainable over time.