Orchestration: Beginner

Play the cron chain variant. extract_orders starts at 01:00 as always, but a slow source database stretches it to 1h47m, so it finishes at 02:47. At 02:00, clean_orders starts on schedule and reads the extract table, which still holds yesterday's rows because tonight's have not been written yet. It cleans yesterday's data and writes the result with tonight's timestamp. At 03:00 join_orders reads that fresh-looking table, at 04:00 aggregate_orders rolls it up, and at 05:05 publish_orders hands the revenue dashboard yesterday's revenue under today's date.

Now play the orchestrator variant. clean_orders has no clock time of its own; its trigger is the success of extract_orders. It waits until 02:47, runs for 20 minutes, and join_orders starts at 03:07. The chain finishes at 03:37, 1h28m earlier than the cron chain managed, and every table it wrote holds today's rows.

Cron is still the right tool for 1 job with no dependents, such as rotating a log file. The trouble starts at the second line that reads what the first line wrote.

A node is a task: a named unit of work with 1 command, such as clean_orders running python clean.py. An edge is a promise about order. The edge from extract_orders to clean_orders says that clean_orders may not start until extract_orders has succeeded. Succeeded means the command exited cleanly; a task that exits with an error does not release the edges leaving it, and a task that is still running has not released them either.

The diagram adds a second root, extract_customers, which depends on nothing. Tasks with no upstream edges become ready at the same moment, so the orchestrator runs the 2 roots side by side when it has 2 workers free. join_orders has 2 upstream edges, so it is released only after both extracts have succeeded; a customer extract that finishes in 4 minutes does not let the join start early.

Orchestrators build the graph from a declaration. The one below uses the shape they share: a name and a schedule, then a list of tasks in which each task names the ones it depends on. No arrow is written anywhere; every edge is implied by a depends_on entry.

Count nodes and edges before running anything. There are 6 tasks. Each name inside a depends_on list is 1 edge pointing into that task. The 2 extracts have 0 and join_orders has 2; clean_orders has 1, as do aggregate_orders and publish_orders. That makes 5 edges. The 2 tasks with empty lists are the roots, and from the lists alone the orchestrator works out the waves in which tasks become ready.

A wave is a convenience for reading, and the orchestrator does not wait for a whole wave. clean_orders starts the moment extract_orders succeeds, even while extract_customers is still running, because clean_orders has 1 upstream edge and that edge is satisfied.

Suppose publish_orders writes a summary table and someone adds publish_orders to the depends_on list of aggregate_orders to read it. Now aggregate_orders waits for publish_orders, which waits for aggregate_orders. Neither can become ready, so the rule that starts a task once its upstream tasks have succeeded finds nothing to start. An orchestrator that reads a declared graph, like the file above, checks for cycles when it parses the declaration and refuses the file before the first night, since a graph with a cycle has no order to run in.

Scheduling creates a run: at 01:00 the orchestrator records that the daily_orders run for 2026-10-22 exists and that none of its tasks have started. Dependency resolution then runs in a loop for the life of the run. Every few seconds the orchestrator asks, for each task not yet started, whether all of its upstream tasks have succeeded, and starts the ones where the answer is yes. The 2 roots pass on the first check; clean_orders passes the moment extract_orders reports success, whether that is 01:40 or 02:47.

Every task attempt gets a row in the orchestrator's database. The row carries the task name with its run date and attempt number, when the attempt started and ended, its final state, and a pointer to its log. The web interface colours each node of the DAG by state, and the run history shows 30 nights of durations. With that record, a question about a bad night has an answer: open the 2026-10-15 run, see extract_orders red with 3 attempts, and read the connection error in the first attempt's log.

The orchestrator issues commands and reads exit codes; the work inside each command happens elsewhere. extract_orders runs a query against the operational database and lands files in the lake. join_orders sends a SQL statement to the warehouse and waits while the warehouse engine runs the join and stores the result. Keep that boundary sharp. A task that pulls 200GB through a worker holds the slot for hours while ready tasks queue behind it, and the worker can run out of memory or disk.

Dagster starts from the tables a pipeline produces rather than the steps that produce them. You declare an asset, such as clean_orders, as a function that returns the data, and name the assets it reads in its arguments; the edges come from those arguments. The emphasis is on knowing which code and inputs produced each table, and when. Dagster also puts weight on running a pipeline on a laptop with typed inputs and outputs before it runs on a schedule, so a broken declaration fails at development time instead of at 01:00.

Cloud providers sell managed versions of all 3. The model is unchanged; what changes is who runs the scheduler and the metadata database.

The orchestrator reads the edges, and the order the tasks were written in plays no part. extract has no upstream edge, so it is the only task ready at 01:00. clean has 1 upstream edge from extract, so it becomes ready when extract succeeds. publish has 1 upstream edge from clean. With 3 tasks and 2 edges there is 1 possible order, and the orchestrator finds it by asking on each loop which task has nothing unfinished above it.

Now the source database drops the connection 12 minutes in. The orchestrator marks attempt 1 failed, waits the 5 minute delay, and starts attempt 2 at 01:17. The whole extract runs again from the first row, succeeds at 01:57, and clean becomes ready. The dashboard is 17 minutes later than usual and correct. Nobody was paged, and the run page shows 2 attempts on extract_orders with the error sitting in the first attempt's log.

If attempts 2 and 3 fail as well, the sequence ends differently. After the third failure the orchestrator marks extract_orders failed and sets clean_orders and publish_orders to upstream-failed without starting either. It then fails the run and sends the alert. The published table keeps yesterday's rows, which is the right outcome: a stale table that everyone knows is stale beats a fresh-looking table built from the wrong night.