dbt vs Airflow

A dbt project is a folder of models, each a SELECT statement in its own file. When 1 model selects from another through ref('stg_orders'), dbt records the dependency, so the project compiles into a graph of models that dbt runs in order. source() does the same for the raw tables an ingestion tool loaded. Nobody maintains a list of steps, because the graph already gives dbt the execution order and table-level lineage, and the docs site is generated from it.

Each model has a materialization that decides what dbt writes: a view, a table rebuilt on every run, an incremental table that only processes new rows, or an ephemeral model inlined into the models that use it. Snapshots record how rows change over time, which is how most dbt projects keep type 2 slowly changing dimensions. dbt build runs models and their tests in graph order, with seeds and snapshots in the same pass, and a failing test makes everything downstream of it skip instead of building on bad data.

The built-in tests (unique, not_null, accepted_values, relationships) check rules you can state in advance. They do not notice a load that arrived 40% short or a column whose values drifted, which is the gap data observability tools fill.

dbt does not extract or load data, doesn't wait for anything outside the warehouse and has no scheduler in its open-source form. It runs SQL, and on some warehouses Python models, on the warehouse's own compute. The dbt tutorial walks through a project from the first model.

2026 changed the product around it. Fivetran and dbt Labs completed their merger in June 2026, and dbt Core 2.0, rebuilt on the Rust foundations of the Fusion engine and still Apache 2.0 licensed, shipped its first alpha in June 2026 and reached release candidate in September 2026. dbt Core 1.12 remains the stable line, and none of this moves dbt into orchestration of steps outside the warehouse.

An Airflow DAG is a Python file that declares tasks and the order between them. A task is an operator configured for 1 job: run a shell command, call an API, submit a Spark job, run a query, start a Kubernetes pod. The scheduler creates a run for each schedule interval or triggering event, hands ready tasks to workers and records each task's state. A task that fails with retries left goes to up_for_retry and runs again after its delay; one whose upstream failed ends upstream_failed without running.

The parts dbt lacks are the point of Airflow. Sensors wait for an external event such as a file landing in object storage. Deferrable operators hand a long wait to the triggerer process so it does not hold a worker slot. Backfills rerun a date range with each run seeing its own interval. When a task fails, the UI shows when it happened and why, and alerts and SLA misses tell someone about it. A guide to Airflow DAGs covers how these pieces are written.

Airflow 3.0, released in April 2025, brought a new UI and DAG versioning. It also moved backfills into the scheduler and added asset-based scheduling; the current release is 3.3.2. Assets let a DAG run when another DAG's task updates a named dataset instead of on a clock. They do not make Airflow aware of data: the Airflow documentation on asset scheduling explains that an asset counts as updated when the task that produces it succeeds, and Airflow never inspects the data itself. What a table contains, and whether it is correct, stays dbt's job.

dbt Core has no scheduler. Something has to call dbt build: a cron entry, a CI workflow such as GitHub Actions, or an orchestrator. For a single warehouse project on a daily cadence, cron plus alerting on the exit code is a legitimate production setup.

The dbt platform has a real job scheduler. A deploy job can run on a cron expression, and the UI also offers simpler interval and specific-hour triggers. On the Starter and Enterprise plans a job can run when another job finishes, which chains jobs across projects. On the Enterprise plans, state-aware orchestration checks each model before it builds and skips the ones whose code and upstream data have not changed.

Every one of those triggers is either a clock or another dbt job. The dbt scheduler can't wait for a file or call an API. It can't run a Spark or Python step outside the warehouse either. Nor can it branch on a condition or retrain a model when a feature table refreshes. When those steps exist, something else must start the dbt job, either through its API or through the dbt Cloud provider for Airflow, and that something is the orchestrator.

dbt on its own scheduler is enough when 4 things hold: all transformation happens in 1 warehouse, a managed ingestion tool lands the raw data on its own schedule, a fixed cadence such as hourly or daily meets the freshness your consumers need, and nothing downstream has to react to the run. Many small analytics teams live here for years, and adding Airflow would give them a metadata database and a fleet of workers to keep upgraded, in exchange for nothing.

You have outgrown dbt's scheduler once you write Python to extract from an API or to parse files before dbt can start. Steps outside the warehouse point the same way. They might run on Spark or in a container; some run on a machine learning platform. A pipeline that must wait for something, such as a partner's file or another team's run, is another sign. Conditional logic counts too, such as a heavy rebuild only on Sundays or skipping a mart when its source didn't arrive. So does wanting retries for steps dbt cannot see, with SLAs and paging when they miss. 1 of these can be patched around; 2 or 3 together are the usual reason teams add Airflow.

The opposite mistake happens too. A team that runs Airflow writes its warehouse transformations as SQL strings inside Python operators. It works until there are dozens of tables, and then the team is rebuilding by hand what dbt already does: dependency order and incremental logic first, then snapshots and tests. That is the point where dbt pays for itself.

A BashOperator that runs dbt build on the Airflow worker is the simplest pattern and the most common first step. It needs dbt and its adapter installed in the Airflow environment, where their Python dependencies can conflict with Airflow's own, and the whole project is 1 task. Splitting the project with selectors, 1 task for the models tagged hourly and another for the daily marts, is the usual first refinement.

The KubernetesPodOperator runs dbt in its own container, which removes the dependency conflict; ECS and Docker have equivalent operators. Because the image pins its own dbt version, each team can upgrade on its own schedule, and the profiles and packages travel in the image too. The trade is start-up time for every pod and artifacts that disappear with the container unless you copy them out.

Teams on the dbt platform keep the job there and let Airflow start it. The dbt Cloud provider for Airflow offers DbtCloudRunJobOperator, which triggers a job and can wait for it in deferrable mode without holding a worker slot, and DbtCloudJobRunSensor, which waits on a run. The catch is 2 schedulers: decide which one owns the timing, or jobs will run twice.

Astronomer Cosmos is open source and currently at version 1.15.1. It parses a dbt Core project and renders it as an Airflow DAG (DbtDag) or a task group inside an existing DAG (DbtTaskGroup), with each model as its own task and its tests run right after it. Airflow then tracks each model separately, so a retry or an alert applies to 1 model. The costs are the parse at DAG load time, a separate dbt start-up for each task in the default local mode, and far more task instances in the metadata database. The watcher execution mode has been stable since 1.15. It runs 1 dbt build and has the per-model tasks follow its progress instead.

Both open-source cores are free under the Apache 2.0 license: dbt Core and Apache Airflow cost nothing to download. You pay for the hosting and the people who operate it. You also pay for the warehouse compute the dbt models consume, which is the same whichever tool triggers them.

The dbt platform prices by seat. Its Developer plan is free for 1 seat and 3,000 successful model builds a month; Starter costs $100 per developer seat per month for up to 5 seats and 15,000 model builds; Enterprise is quoted. A team of 4 analytics engineers on Starter pays $400 a month, scheduler included.

Managed Airflow prices by environment. On Amazon MWAA a small environment costs $0.49 an hour in US East, which over 730 hours is $357.70 a month before extra workers and metadata storage. Self-hosting removes that bill and replaces it with engineering time: you operate every component yourself, from the scheduler to the metadata database, and you own the upgrades. The move from Airflow 2 to 3 was a migration with breaking changes, not a version bump.

Set side by side, the license line rarely decides it. The deciding cost is the pipeline you can't build without an orchestrator, or the orchestrator you run for a pipeline that never needed one.

Job postings are the cleanest evidence that dbt and Airflow are complements. The site tags every active data engineering posting with the tools it names, and the 2 names travel together far more often than chance would put them on the same page.

Airflow is named on 7.5% of the postings that name any tool, 2,756 of them, and dbt on 5.2%, 1,905. Of the postings that ask for dbt, 50% also ask for Airflow, and 35% of the Airflow postings ask for dbt. That pairing is 5.2 times as common as it would be if the 2 tools were listed independently.

The newer orchestrators have not displaced Airflow in hiring. Even postings that name Dagster mention Airflow 79% of the time, usually because the team runs both or is migrating between them. For a data engineer the practical reading is to learn both: dbt for how the warehouse layer is modeled and tested, Airflow for how every step around it is scheduled and recovered.

Dagster is the orchestrator most often weighed against Airflow for dbt work, because it models pipelines as assets rather than tasks and represents each dbt model as its own asset. The dbt graph and the orchestrator graph become 1 graph, so each model's status and freshness show up in the orchestrator, lineage included, with no translation layer such as Cosmos. The trade is fewer integrations than Airflow's providers offer and a mental model the team has to learn.

Prefect writes flows as decorated Python functions and suits teams whose pipelines are mostly Python. Warehouse-native schedulers, such as the task schedulers built into the big cloud warehouses and lakehouse platforms, can run dbt when every step lives on that platform.

For a new platform with no orchestrator yet, the choice between Airflow and Dagster is real and worth a proof of concept on your own dbt project. For a team with a working Airflow deployment, Airflow 3 assets plus Cosmos close most of the gap, and a migration rarely pays for itself.

dbt is where most teams encode their dimensional model, and snapshots are the clearest case of the 2 tools depending on each other. A snapshot compares the current source rows with the last recorded version and writes a new version with dbt_valid_from and dbt_valid_to whenever a row changed, detected by an updated_at timestamp or by comparing chosen columns. That is a type 2 slowly changing dimension with the merge logic written for you.

A snapshot only records the changes it sees, so its schedule is part of its correctness. A customer record that changes twice between 2 snapshot runs keeps only the second change. That's why snapshots often run more often than the nightly models built on them, which is a cadence decision that belongs in the orchestrator, not in the SQL.

The production questions that follow are orchestration questions too: how fast the snapshot table grows, how you backfill it when a source was wrong for a week, and how you rebuild a dimension without losing history. None has a dbt-only answer.

Interviewers rarely ask which tool is better. They give a scenario and listen for whether you keep transformation and orchestration in separate layers. A common opener is to design the pipeline behind a set of dashboards: a strong answer names the ingestion tool, the orchestrator, the dbt layers from staging to marts, where freshness checks live and who is paged when a source is late.

Failure questions follow. What happens when 1 model fails halfway through a run, and what does the retry rerun? Explaining that a whole-project retry of a 6 model project can rebuild 4 models that had already passed, and how to avoid it, is exactly the reasoning they want. Late-arriving events are another favorite: an incremental model that filters on the maximum loaded timestamp misses rows that arrive late, and the fix is a lookback window with a merge on a unique key, sized against how late the data really arrives.

These scenarios sit in the system design round of a data engineering interview. To prepare, work through the data pipeline interview questions, the orchestration-specific Airflow interview questions, and the data pipeline practice problems, which make you build the transformations an orchestrator would run.