Why Cron Is Not an Orchestrator
Concepts covered: DAG Orchestration
A schedule is a list of start times
Cron is a program that lives on a server and starts a command at a clock time. A cron entry has 2 parts: when, written as 5 time fields, and what, the command to run. That is the whole contract. Cron does not read the command's output, does not wait for it to finish, and does not know that the entry on the next line depends on the one above.
A nightly orders pipeline built on cron looks like 5 entries in 1 file. Each entry starts an hour after the previous one, because an hour is longer than any of these jobs has ever taken.
| Clock time | Cron fields | Job | Usual duration |
|---|---|---|---|
| 01:00 | 0 1 * * * | extract_orders | 40 min |
| 02:00 | 0 2 * * * | clean_orders | 20 min |
| 03:00 | 0 3 * * * | join_orders | 15 min |
| 04:00 | 0 4 * * * | aggregate_orders | 10 min |
| 05:00 | 0 5 * * * | publish_orders | 5 min |
The hour-long gaps are a guess about durations, and the guess holds until the first night it does not.
- extract_orders (transform, cron)
- clean_orders (transform, cron)
- join_orders (transform, cron)
- aggregate_orders (transform, cron)
- publish_orders (transform, cron)
- revenue_dashboard (consumer, Looker)
- extract_orders -> clean_orders
- clean_orders -> join_orders
- join_orders -> aggregate_orders
- aggregate_orders -> publish_orders
- publish_orders -> revenue_dashboard
- Cron chain: Result: today's revenue shows yesterday's data under today's name. Every job succeeded
- Orchestrator: Result: today's revenue has today's data at 03:37
Reading the night step by step
Play the cron chain variant. extract_orders starts at 01:00 as always, but a slow source database stretches it to 1h47m, so it finishes at 02:47. At 02:00, clean_orders starts on schedule and reads the extract table, which still holds yesterday's rows because tonight's have not been written yet. It cleans yesterday's data and writes the result with tonight's timestamp. At 03:00 join_orders reads that fresh-looking table, at 04:00 aggregate_orders rolls it up, and at 05:05 publish_orders hands the revenue dashboard yesterday's revenue under today's date.
Every job exited with status 0, and no log line says error. The number on the dashboard is plausible, because yesterday's revenue looks a lot like today's. This is the canonical cron failure: the chain runs to completion on stale input, and the first sign of it is a question from finance 3 days later.
Now play the orchestrator variant. clean_orders has no clock time of its own; its trigger is the success of extract_orders. It waits until 02:47, runs for 20 minutes, and join_orders starts at 03:07. The chain finishes at 03:37, 1h28m earlier than the cron chain managed, and every table it wrote holds today's rows.
Failure modes of a cron chain
What an orchestrator adds
An orchestrator keeps the clock time and adds a graph. Its jobs are now called tasks, and it stores the edges between them under 1 rule: a task starts when the tasks it depends on have all succeeded. It retries a task that fails, and it keeps a record of every run, so the question of what fed the 2026-10-22 dashboard has an answer. Cron supplies the clock and none of the rest.
- Cron knows
- The current time
- The command on this line
- Nothing about the line above or below
- An orchestrator knows
- The current time and the schedule
- Which tasks this task depends on
- Whether those tasks succeeded, and when
- How many times to retry a failure
- The history of every run
Cron is still the right tool for 1 job with no dependents, such as rotating a log file. The trouble starts at the second line that reads what the first line wrote.
On the cron chain above, extract_orders crashes at 01:10 with a database connection error and writes nothing. What does clean_orders do at 02:00?
- It waits until someone reruns the extract, because cron queues entries behind a failed one.
- It starts at 02:00 and reads the extract table, which still holds yesterday's rows, so the chain publishes stale data with a fresh timestamp.
- It fails immediately because its input table is empty after the crash.
- It is skipped because cron saw the non-zero exit code of the extract.
About this lesson
This section is part of the Orchestration: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.