Apache Airflow vs Dagster

Both versions are plain Python. In Airflow, @dag carries the start date and a schedule with its catch-up policy, and the tasks sit inside it. Order comes from calling one task with another's return value. Airflow stores that returned value through XCom, its mechanism for passing small values between tasks, so large datasets still belong in external storage, with the task passing a path or a key.

In Dagster, the @dg.asset function is the unit, and its name is the asset's name. A parameter called raw_orders tells Dagster this asset depends on the asset of that name, which makes the dependency graph part of the function signature. Where an asset needs another asset to exist but does not read its value, deps=[...] declares the dependency without passing data. Nothing in the file says when it runs: schedules and sensors are attached separately, along with partitions, so the same asset can be rebuilt on a cron or an event, or run by hand.

Airflow began with time-based schedules and has added data-aware ones. Asset-aware scheduling arrived in 2.4: a producer task lists an asset as an outlet, and a consumer Dag names that asset in its schedule. The Airflow asset scheduling docs add 2 details that matter in practice. An asset counts as updated only when the producing task succeeds, and a consumer waiting on several assets runs once all of them have updated since its last run. Expressions built with & and | combine them.

Airflow April 22, 2025 brought 3.0, which the project's release post describes as adding event-driven scheduling from outside systems, DAG versioning so a run finishes on the version it started with, and backfills run by the scheduler and started from the UI or API. External events arrive by REST push or by an asset watcher that polls a source.

Dagster puts the asset first and the trigger second. A schedule runs assets at an interval, a sensor triggers them when an external event occurs, such as a file landing, and both can carry run configuration. Because the asset is the primary object, the UI can show for every table when it was last built and what it was built from, without extra declarations.

Reprocessing history is where the models differ most in daily use. In Dagster, a partition is a slice of an asset, usually a time window. Attach partitions_def=DailyPartitionsDefinition(start_date=...) to an asset and the run reads context.partition_key to know which day it is building. The docs recommend keeping each asset to 100,000 partitions or fewer for the UI to stay responsive, and multi-partition runs need the daemon running to queue them.

In Airflow, each Dag run carries a data interval and a task reads its logical date to know which slice to build. Before 3.0, backfills were started from the command line. Since 3.0 the scheduler runs them. Either way, the discipline that makes a backfill safe is the same in both tools: write each run so that rerunning a day replaces that day instead of appending to it, which is the idempotency property data engineering interviews test.

Dagster's asset checks add data quality tests to the asset itself. A check decorated with @dg.asset_check runs after the asset is materialized, and by default a failed check does not stop downstream assets unless it is marked blocking=True. In Airflow, the equivalent is a task you place in the Dag, or an external tool such as a data quality framework that you call from a task.

Airflow runs as a set of services around a metadata database. The scheduler is a persistent service, and running more than 1 is supported on PostgreSQL 12+ and MySQL 8.0+. Work runs on an executor. The executor docs list Local as the default and also Celery, Kubernetes, Edge, ECS and Batch executors, and since 2.10 you can configure several at once and choose per task or per Dag.

Dagster's open source deployment has 3 long-running pieces: a webserver for the UI and GraphQL queries, a daemon that runs schedules and sensors and queues runs, and a code location server for each project of definitions. The webserver can have replicas. The daemon cannot, and each code location gets 1 server, which is worth knowing before you promise high availability. The executor and run launcher are swappable, and the default SQLite run storage is normally swapped for Postgres in production.

Code locations are the operational idea Dagster leans on: each team ships its own definitions as a separate deployable, so 1 team's dependency upgrade does not break another's. Airflow reaches the same isolation through separate deployments, per-task containers on the Kubernetes executor, or the Task SDK in 3.0, which separates the code a task runs from the Airflow core.

Airflow fits when the work is a set of jobs more than a set of tables. Running a nightly export, a model training job, a call to a vendor API and a cleanup in a fixed order is naturally a Dag of tasks. Airflow's operators and providers cover a large number of systems out of the box.

It fits when you already run it. Existing Dags and shared operators are an asset of their own, as are runbooks and on-call knowledge, and Airflow 3 added scheduler-run backfills, DAG versioning and asset-based triggers, so some of the reasons teams look elsewhere have been addressed inside it. It also fits when hiring or interviewing matters: Airflow knowledge is the more common requirement, and the Airflow interview questions cover what candidates are asked.

Dagster fits when the pipeline is mostly tables and models and the questions people ask are about the data: what is stale, what feeds this dashboard, what must rebuild if this source changes. Because assets and their dependencies are the primary objects, the definitions already give you lineage, freshness and partition status without extra bookkeeping.

It fits when you start fresh and want local development and testing to be a first-class path: assets are plain functions that can be called in a test, and resources make external connections a swappable parameter. It also fits teams with many partitioned datasets, where seeing which days are built and rebuilding a range from the UI is a daily task.

Switching orchestrators does not have to be a rewrite. Dagster's dagster-airlift connects to Airflow's REST API, loads each Airflow Dag into Dagster as an asset definition and adds a sensor that polls Airflow for execution history. In its observe stage you map Airflow Dag and task IDs to the Dagster definitions they produce, with no change to the Airflow code.

In the migrate stage you move execution of individual tasks to Dagster assets in any order, and Airflow proxies each migrated task to Dagster. The task-level migration tutorial in the Dagster docs notes that you can roll a task back by changing a single line in the Airflow Dag, and recommends writing acceptance tests in Dagster before moving a task. This path also suits teams that want Dagster's asset view over an Airflow estate they have no plan to retire.

The other direction has no equivalent tool in the docs reviewed for this page, so a move from Dagster to Airflow is a rewrite of the orchestration layer, with the data transformations themselves usually reusable.

Interviewers rarely care which tool you picked. They ask how you would keep a pipeline correct when runs fail, and when they arrive late or repeat: how you make a task idempotent, how you backfill 30 days without double counting, what you alert on and how you order dependencies. Those answers are the same in both tools, and naming the Airflow or Dagster feature that implements each one makes them concrete.

Practice the design questions in the data pipeline interview questions, build the pipelines yourself in the data pipeline practice problems, and rehearse explaining a scheduler choice for the system design round. For more tools in the same space, see data orchestration tools, the Airflow DAG guide and dbt vs Airflow.