The Major Orchestrators by Name
Concepts covered: DAG Orchestration
Most open source orchestration runs on 3 tools, and each gives a different answer to the question of what a pipeline author should write down first. All 3 run the model from the previous section underneath.
Airflow: tasks and operators
Airflow came out of Airbnb's data team, where a nightly set of cron jobs had grown past what anyone could reason about, and it has been an Apache project since. Its unit is the task, declared in Python inside a DAG object, and its main draw is a library of operators: prebuilt task types, such as one that runs a SQL statement and one that waits for a file to appear, with others that call APIs. Airflow emphasizes breadth. If a system exists, there is probably an operator for it. Most engineers have also already seen its scheduler and web interface.
- 2015
- 2019
Dagster: assets first
Dagster starts from the tables a pipeline produces rather than the steps that produce them. You declare an asset, such as clean_orders, as a function that returns the data, and name the assets it reads in its arguments; the edges come from those arguments. The emphasis is on knowing which code and inputs produced each table, and when. Dagster also puts weight on running a pipeline on a laptop with typed inputs and outputs before it runs on a schedule, so a broken declaration fails at development time instead of at 01:00.
Prefect: plain Python functions
Prefect treats a workflow as a Python function, called a flow, that calls other functions, called tasks. The graph is whatever the function does when it runs, so a flow can loop over a list of 40 regions and create 40 tasks at run time without declaring each in advance. The emphasis is on writing the pipeline as ordinary code and letting the orchestrator wrap it with retries and state records as it runs.
The shared model
Side by side, the 3 use different words for the same nouns. Each has a container for a graph, a node that does work, a schedule that creates runs, a retry policy per node, and a run history behind a web interface. Learn the model once and the first week on any of the 3 is vocabulary.
| Concept | Airflow calls it | Dagster calls it | Prefect calls it |
|---|---|---|---|
| Graph container | DAG | job, or the asset graph | flow |
| Unit of work | task | op, or asset | task |
| One execution | DAG run | run | flow run |
| When runs begin | schedule on the DAG | schedule on the job | schedule on the deployment |
Under the declarations the 3 ship the same moving parts. A scheduler process creates runs and resolves dependencies, a pool of workers executes commands, and a metadata database holds every state change; the web interface is a window onto that database. On any of the 3, if the metadata database is down, no run starts.
What differs is the thing you write first. Airflow asks for the task and its order, Dagster asks for the table and its inputs, Prefect asks for the function. A pipeline designed as a graph of tasks with a schedule and retry counts moves between them with its design intact; what changes is the file that declares it.
If your company already runs one of them, use that one; a second orchestrator doubles the number of places a run can hide. For a team starting fresh, Airflow's operator library covers the most systems out of the box, Dagster's asset view answers the question of what made a table without extra work, and Prefect keeps the declaration closest to the Python the team already writes.
Cloud providers sell managed versions of all 3. The model is unchanged; what changes is who runs the scheduler and the metadata database.
Your team moves the daily_orders pipeline from one of the 3 orchestrators to another. Which part of the design can you keep without a redesign?
- Only the SQL inside each task; the graph, schedule and retries must be rethought from scratch for the new system.
- Nothing, because each orchestrator runs a different model and the pipeline must be rebuilt as a new kind of object.
- Only the schedule, since every orchestrator reads the same cron expression and nothing else is shared.
- The graph of 6 tasks and 5 edges, the 01:00 schedule and the retry count per task, each re-declared in the new orchestrator's vocabulary.
About this lesson
This section is part of the Orchestration: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.