What an Orchestrator Does
Concepts covered: DAG Orchestration
Strip an orchestrator down to its duties and 4 remain. Each answers a question cron left open, and each concerns control of the work; other systems handle the data itself.
| Responsibility | Question it answers | In the orders pipeline |
|---|---|---|
| Scheduling | When does a run begin? | A run starts at 01:00 every day |
| Dependency resolution | Which task may start now? | join_orders starts when clean_orders and extract_customers have both succeeded |
| Failure handling | What happens when a task exits with an error? | Retry twice, 5 minutes apart; then mark it failed and hold everything downstream |
| State and visibility | What ran, when, for how long, with what result? | The 2026-10-22 run page shows extract_orders took 1h47m and the rest waited |
Scheduling and dependency resolution
Scheduling creates a run: at 01:00 the orchestrator records that the daily_orders run for 2026-10-22 exists and that none of its tasks have started. Dependency resolution then runs in a loop for the life of the run. Every few seconds the orchestrator asks, for each task not yet started, whether all of its upstream tasks have succeeded, and starts the ones where the answer is yes. The 2 roots pass on the first check; clean_orders passes the moment extract_orders reports success, whether that is 01:40 or 02:47.
Failure handling
A task fails when its command exits with a non-zero code or runs past its timeout. The orchestrator then applies the task's retry policy: wait the retry delay, run the same command again, up to the retry count. During the retries every downstream task keeps waiting, which cron could never do. When the last retry fails, the task is marked failed and every task downstream of it is marked upstream-failed without running. An alert goes out only if the DAG declares a target for it, such as an email address or a chat channel.
A retry starts the command from the beginning, because the orchestrator has no idea what the first attempt wrote before it died. A task that appended 10,000 rows and then crashed will append them again on the retry. Writing tasks so that a second run produces the same result as a first run is its own subject, covered in a later lesson; for now, know that the orchestrator provides the retry and the task has to be written so that a rerun does no harm.
State and visibility
Every task attempt gets a row in the orchestrator's database. The row carries the task name with its run date and attempt number, when the attempt started and ended, its final state, and a pointer to its log. The web interface colours each node of the DAG by state, and the run history shows 30 nights of durations. With that record, a question about a bad night has an answer: open the 2026-10-15 run, see extract_orders red with 3 attempts, and read the connection error in the first attempt's log.
What it hands to other systems
The orchestrator issues commands and reads exit codes; the work inside each command happens elsewhere. extract_orders runs a query against the operational database and lands files in the lake. join_orders sends a SQL statement to the warehouse and waits while the warehouse engine runs the join and stores the result. Keep that boundary sharp. A task that pulls 200GB through a worker holds the slot for hours while ready tasks queue behind it, and the worker can run out of memory or disk.
| Work | Owner |
|---|---|
| Deciding that a run starts at 01:00 | Orchestrator |
| Deciding that join_orders may start | Orchestrator |
| Running the SELECT that joins orders to customers | Warehouse engine |
| Holding the joined table between runs | Warehouse storage |
| Moving bytes from the operational database into the lake | The extract program |
| Running the extract a second time after a timeout | Orchestrator |
| Recording that attempt 2 succeeded at 01:57 | Orchestrator |
Which of these is the orchestrator's own job, rather than work it delegates to another system?
- Executing the join between the cleaned orders and the customers table.
- Compressing the extracted files before they land in the lake.
- Deciding that join_orders may start because clean_orders and extract_customers have both succeeded.
- Keeping the joined table on disk so that aggregate_orders can read it.
About this lesson
This section is part of the Orchestration: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.