First DAG: 3 Tasks, 1 Schedule

Concepts covered: DAG Orchestration

The pipeline is the smallest version of the orders chain: extract, clean, publish, started at 01:00 with 1 retry policy for every task. The code below is illustrative. It has a name and a schedule, the retry policy, and 3 tasks joined by 2 edges, which is the shared shape; each orchestrator spells those same parts in its own syntax.

from orchestrator import DAG, task

with DAG(
    dag_id="daily_orders",
    schedule="0 1 * * *",
    retries=2,
    retry_delay_minutes=5,
) as dag:
    extract = task("extract_orders", command="python extract.py")
    clean = task("clean_orders", command="python clean.py")
    publish = task("publish_orders", command="python publish.py")

    extract >> clean >> publish

Reading it line by line

What each line declares

• schedule="0 1 * * *": the 5 cron fields. Minute 0 of hour 1, on every day of every month, whatever the weekday. One run per night at 01:00.
• retries=2: each task gets 3 attempts in total, the first run plus 2 retries, and the policy on the DAG applies to every task in it.
• retry_delay_minutes=5: the orchestrator waits 5 minutes between a failed attempt and the next one.
• task(...): a named node with 1 command. The name is what the run page and the alert will show, so it names the output.
• extract >> clean >> publish: 2 edges. Read >> as 'must succeed before': the task on the left must succeed before the task on the right may start.

The order of execution

The orchestrator reads the edges, and the order the tasks were written in plays no part. extract has no upstream edge, so it is the only task ready at 01:00. clean has 1 upstream edge from extract, so it becomes ready when extract succeeds. publish has 1 upstream edge from clean. With 3 tasks and 2 edges there is 1 possible order, and the orchestrator finds it by asking on each loop which task has nothing unfinished above it.

You add a fourth task, extract_customers. publish_orders reads its output, but clean_orders does not. Which single change starts every task as soon as its inputs exist?

• Add extract_customers >> clean, so customer data lands before cleaning starts.
• Add publish >> extract_customers, so the customer extract follows the publish.
• Add extract_customers >> publish, so publish waits on both clean and the customer extract.
• Give extract_customers a cron entry at 00:30, so it finishes before the 01:00 run.

A night with a failure

Now the source database drops the connection 12 minutes in. The orchestrator marks attempt 1 failed, waits the 5 minute delay, and starts attempt 2 at 01:17. The whole extract runs again from the first row, succeeds at 01:57, and clean becomes ready. The dashboard is 17 minutes later than usual and correct. Nobody was paged, and the run page shows 2 attempts on extract_orders with the error sitting in the first attempt's log.

If attempts 2 and 3 fail as well, the sequence ends differently. After the third failure the orchestrator marks extract_orders failed and sets clean_orders and publish_orders to upstream-failed without starting either. It then fails the run and sends the alert. The published table keeps yesterday's rows, which is the right outcome: a stale table that everyone knows is stale beats a fresh-looking table built from the wrong night.

Do

• Name tasks after what they produce, so the run page reads as the pipeline
• Give every task a retry count, even 1, so a flaky connection does not page anyone
• Let the edges decide order; never encode order in start times
• Read the run page after the first night and note each task's duration

Avoid

• Chain tasks by giving each its own clock time inside one DAG
• Set so many retries that a broken source gets queried again and again for hours
• Move data through the orchestrator's own workers
• Put 2 unrelated pipelines in 1 DAG because they share a schedule

Capstone

extract_orders succeeds at 01:40. clean_orders then fails on all 3 attempts because a column it expects is missing. At 02:30, what is the state of publish_orders, and what does the dashboard show?

• publish_orders ran at 02:30 as scheduled and published the uncleaned extract, so the dashboard shows raw rows.
• publish_orders is marked upstream-failed and never ran; the dashboard shows the previous night's published table, and the run page is red at clean_orders.
• publish_orders is retrying, because the retry policy on the DAG applies to the run as a whole.
• publish_orders succeeded, because the orchestrator skips a failed upstream task once its retries are spent.

About this lesson

This section is part of the Orchestration: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.