Data Orchestration Tools

The first choice is what the graph describes. A task-centric orchestrator models the steps: run this extract, then this load, then this check. An asset-centric orchestrator models the things the steps produce: this table exists, this file is current, and the system works out which code must run to make them so.

Dagster is built on the second view. Its documentation defines an asset as an object in persistent storage, such as a table or file, or a persisted machine learning model, and materializing an asset as running its function and saving the result. Dependencies live between assets, so the lineage graph and the run graph are the same graph.

Airflow began on the first view and now reaches toward the second. A task declares the asset it updates in outlets, and a Dag declares the assets it waits for in schedule. Per the Airflow asset scheduling documentation, a Dag scheduled on a list of assets runs after every listed asset has updated since its last run, and an asset counts as updated only when its task succeeds. Asset-based scheduling is one of the headline areas of the Airflow 3.0 release.

Neither view is more correct. Task-centric graphs fit pipelines whose steps don't produce a named dataset, such as calling an API or starting a cluster. Asset-centric graphs fit pipelines whose steps are mostly the construction of tables and files, where the useful question is whether this table is current and what is downstream of it.

Airflow describes itself as an open-source platform for developing workflows in Python and monitoring their scheduled runs, and its documentation calls it a platform for orchestrating batch workflows. Its 3.0 release came on 2025-04-22, and its release notes list 3.3.2 as the newest version, from 2026-09-17.

Airflow 3 changed its internals. A Dag processor parses Dag files and serializes them into the metadata database. An API server serves the REST API and the interface, and tasks report state through it instead of reaching the database directly. Airflow needs a scheduler, a Dag processor, an API server and a metadata database on PostgreSQL or MySQL; workers and a triggerer are optional. Knowing that list matters when you run it yourself, because each one is a component to size and upgrade, and one more thing to monitor.

Airflow fits when the pipeline touches many different systems. Its provider packages cover a long list of services, so a team that moves data between a warehouse and object storage, or from SaaS APIs into a cluster, usually finds an operator for each. It also fits when a managed option matters, because Amazon and Google both run it as a service.

It fits less well when most of the graph is tables a team wants to inspect by name, or when a team wants pipelines defined in something lighter than a Dag file. For the mechanics, see what an Airflow DAG is and how it runs, and for the questions interviewers ask about it, the Airflow interview questions.

Dagster calls itself a data orchestrator built for data engineers, with lineage, observability, a declarative programming model and testability built in. Its repository describes it as an orchestration platform for developing and observing data assets in production, and it is released under the Apache 2.0 licence.

Because the unit is the asset, the questions an on-call engineer asks are first-class screens: which assets are stale, which run produced this table, and which assets sit downstream of the one that broke. A pipeline is written as Python functions marked as assets, with dependencies declared between them, so a small pipeline is a small file.

Dagster fits when the pipeline is mostly the construction of datasets, when a team wants checks and lineage attached to the same objects the schedule builds, and when engineers want to run and test pipeline code locally as ordinary Python. Dagster+ is the vendor's managed offering with a hosted Dagster instance. It fits less well for orchestration that is mostly steps with no dataset to point at, where modelling everything as an asset adds ceremony.

Prefect 3 describes itself as an open-source orchestration engine that turns Python functions into production-grade data pipelines. Flows and tasks are ordinary functions with decorators, and there is no DSL or YAML to learn. Prefect 2.0 removed the requirement that a workflow be an explicit DAG, so dynamic fan-out and other loops or branches are written as normal Python control flow.

Its execution model is the part that most often decides the choice. A work pool is the bridge between the orchestration layer and the infrastructure where flows run. In the hybrid model you run a worker in your own environment that polls the pool and submits runs to your infrastructure. Flow code and data stay on your side. Push pools hand runs directly to a serverless provider such as AWS ECS or Google Cloud Run with no worker, and managed pools run on Prefect's own infrastructure.

Prefect fits when a team wants to turn existing Python scripts into monitored, retried, scheduled pipelines with the least rewriting, and when runs must start in different kinds of infrastructure. Prefect 3.0 also open-sourced the events and automations backend, so a run can start from an external event, not only a clock. It fits less well when a team wants every system it touches to come with a ready-made operator, since the unit of work is a Python function you write.

Kestra describes itself as an open-source orchestration platform, and its repository calls it an event-driven platform with declarative YAML flows. A flow is a YAML document with an id and a namespace, followed by a list of tasks, each task naming a plugin type. The same flow can be edited as YAML, in a no-code editor or with an AI assistant, and every editor changes the same document.

The trade is clear. A flow is easy to read for someone who does not write Python, and it can sit in Git beside infrastructure definitions. The cost is that logic that is natural in a language, such as loops with computed inputs or shared helper functions, has to be expressed through the plugin vocabulary. Kestra is released under the Apache 2.0 licence, and an Enterprise Edition adds access control and policies, and separate worker groups.

Kestra fits when pipeline owners include analysts and platform engineers who read YAML more comfortably than Python. It also fits when the same orchestrator has to schedule non-data work, such as infrastructure tasks or business processes. It fits less well for teams whose pipeline logic is mostly Python that they want to unit test as code.

Some orchestrators belong to the platform where the work already runs. AWS Step Functions models a workflow as a state machine whose steps are states, and it calls AWS services directly, including Glue, EMR, Athena and Lambda. A Standard workflow gives exactly-once execution, can run for up to 1 year and is priced by state transition. An Express workflow is built for high event rates, gives at-least-once execution, runs for up to 5 minutes and is priced by executions and duration. The Step Functions documentation lists both limits.

Argo Workflows is a container-native workflow engine for orchestrating parallel jobs on Kubernetes, implemented as a Kubernetes custom resource, with each step a container and the workflow declared as a DAG or as steps. It is a graduated project of the Cloud Native Computing Foundation. It fits teams that already run Kubernetes and want pipelines to be Kubernetes objects.

Lakeflow Jobs is workflow automation inside Databricks. A job coordinates tasks, which can be notebooks and pipelines as well as Python scripts, supports if/else branching and for-each loops, starts on a schedule or when new data arrives in cloud storage, and can include dbt. The documentation sets a limit of 1,000 tasks per job and 2,000 concurrent task runs per workspace.

These 3 share a trade. Inside their own platform they need almost no setup, with no scheduler to host and native access to the platform's identity and logging. Outside it they get awkward, because a pipeline that spans a warehouse and a SaaS API, and another cloud besides, has to cross the platform boundary in every step. They fit when most of the work lives in 1 platform, and a general-purpose orchestrator is the better pick when it does not.

The operating model matters as much as the tool. Self-hosting an orchestrator means running its scheduler and workers against a metadata database, and being on call for all of it, upgrades included. For a small team the orchestrator can become a second system to run beside the pipelines it schedules.

Managed services remove most of that. Amazon MWAA runs Airflow, so you don't manage or secure the infrastructure behind its scalability and availability. The scheduler and workers run as Fargate containers, and AWS manages the metadata database. Google's managed Airflow service, Cloud Composer, provisions an environment on Google Kubernetes Engine with the web server and the metadata database, plus a Cloud Storage bucket for Dag files and logs. Dagster+, Prefect Cloud and Kestra's Enterprise Edition give the same relief for their own tools.

The cost of managed is lag and control: a managed service offers the versions its provider has released, with its own limits on configuration and networking. A team with strict network isolation or custom plugins, or one that needs to upgrade on its own schedule, may still choose to host, and a team with 2 engineers and 40 pipelines usually should not.

Searches for the best orchestration tool for data pipelines expect 1 answer, but the answer depends on 4 questions. Each one rules tools out before any feature comparison.

The first question is what the graph is made of. Mostly tasks across many systems points to Airflow or Prefect, or to Kestra. Mostly tables and files points to Dagster, or to Airflow's asset scheduling. The second is where the work runs: one platform points to that platform's engine, and many systems point to a general-purpose tool. Who writes and reads the pipelines comes next. Python-first engineers suit Airflow and Dagster, and Prefect too, while YAML-first owners suit Kestra. Last, who operates it? A team that cannot staff the control plane should price the managed option for any tool before comparing them.

Then test the shortlist against your worst pipeline, not the tutorial one. Write the pipeline with the longest dependency chain and the most awkward trigger in each candidate, rerun a failed middle step, and backfill a month of history. The tool that makes those 3 jobs plain is the one your team will still like in the second month.

An orchestrator does not transform data. dbt runs SQL transformations inside a warehouse in dependency order, so it covers the transform step and leaves ingestion and cross-system coordination to something else. The comparison is covered in dbt vs Airflow, and most teams run both: the orchestrator triggers dbt after the load finishes.

It is also not a streaming engine. Airflow's documentation describes it as a platform for orchestrating batch workflows, and continuous event processing belongs to a stream processor that keeps running. The orchestrator's job there is to deploy the stream job and watch it. The difference between the 2 models is laid out in batch vs streaming data pipelines.

And it isn't an ingestion or ETL product. Those tools move and reshape rows. The orchestrator decides when they run and what happens when they fail, which is why choosing an ETL tool is a different question from the one on this page.

Interviewers rarely ask you to name the best tool. They ask how you would run a pipeline you designed: what triggers it, what happens when step 3 of 5 fails, how you rerun it without double counting, how you backfill 30 days, and how you know it is late before a stakeholder does. A strong answer names the tool you would use, states the trade you accepted, and says what would make you choose another.

That ground sits inside data pipeline interview questions, and you can design the pipelines yourself with the data pipeline practice problems. The design round puts orchestration in context: the system design round asks where scheduling and retries sit in the whole architecture, and where monitoring does.