The DAG: Tasks, Edges, No Cycles

Concepts covered: DAG Orchestration

The graph an orchestrator holds has a name: a directed acyclic graph, DAG for short. Each of the 3 words rules out one thing, and the things ruled out are what make the graph runnable.

Nodes are tasks, edges are must-finish-before

A node is a task: a named unit of work with 1 command, such as clean_orders running python clean.py. An edge is a promise about order. The edge from extract_orders to clean_orders says that clean_orders may not start until extract_orders has succeeded. Succeeded means the command exited cleanly; a task that exits with an error does not release the edges leaving it, and a task that is still running has not released them either.

The diagram adds a second root, extract_customers, which depends on nothing. Tasks with no upstream edges become ready at the same moment, so the orchestrator runs the 2 roots side by side when it has 2 workers free. join_orders has 2 upstream edges, so it is released only after both extracts have succeeded; a customer extract that finishes in 4 minutes does not let the join start early.

Reading a DAG declaration

Orchestrators build the graph from a declaration. The one below uses the shape they share: a name and a schedule, then a list of tasks in which each task names the ones it depends on. No arrow is written anywhere; every edge is implied by a depends_on entry.

dag: daily_orders
schedule: "0 1 * * *"
tasks:
  - name: extract_orders
    depends_on: []
  - name: extract_customers
    depends_on: []
  - name: clean_orders
    depends_on: [extract_orders]
  - name: join_orders
    depends_on: [clean_orders, extract_customers]
  - name: aggregate_orders
    depends_on: [join_orders]
  - name: publish_orders
    depends_on: [aggregate_orders]

Count nodes and edges before running anything. There are 6 tasks. Each name inside a depends_on list is 1 edge pointing into that task. The 2 extracts have 0 and join_orders has 2; clean_orders has 1, as do aggregate_orders and publish_orders. That makes 5 edges. The 2 tasks with empty lists are the roots, and from the lists alone the orchestrator works out the waves in which tasks become ready.

A wave is a convenience for reading, and the orchestrator does not wait for a whole wave. clean_orders starts the moment extract_orders succeeds, even while extract_customers is still running, because clean_orders has 1 upstream edge and that edge is satisfied.

Why no cycles

Suppose publish_orders writes a summary table and someone adds publish_orders to the depends_on list of aggregate_orders to read it. Now aggregate_orders waits for publish_orders, which waits for aggregate_orders. Neither can become ready, so the rule that starts a task once its upstream tasks have succeeded finds nothing to start. An orchestrator that reads a declared graph, like the file above, checks for cycles when it parses the declaration and refuses the file before the first night, since a graph with a cycle has no order to run in.

An engineer wants tonight's extract_orders to wait until tonight's publish_orders has finished, so the dashboard is never mid-update while the next extract runs. They add publish_orders to the depends_on list of extract_orders. What happens?

• The orchestrator refuses to load the DAG, because the new edge closes a cycle: extract waits on publish, which waits, through 4 tasks, on extract.
• The DAG loads, and tonight's extract waits until tonight's publish finishes, exactly as intended.
• The orchestrator loads the DAG and runs publish_orders first, since it now has an outgoing edge to the root.
• The DAG loads, but extract_orders runs twice per night, once before publish and once after it.

About this lesson

This section is part of the Orchestration: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.