Schedules: Cron, Interval, Event
A schedule tells the orchestrator when to start a new run of a DAG. Nearly every pipeline uses one of 3 kinds: a cron schedule names moments on the wall clock, an interval schedule names a fixed gap between runs, and an event schedule names a signal, such as a file landing, and starts a run each time the signal arrives.
| Schedule | Written as | Starts runs at | A run starts because |
|---|---|---|---|
| Cron | `0 6 * * *` | 06:00 every day | the clock reached a named minute |
| Cron | `*/15 * * * *` | :00, :15, :30 and :45 | the clock reached a named minute |
| Interval | every 15 minutes from 10:07 | 10:07, 10:22, 10:37 | a fixed gap passed since the anchor |
| Event | on file `rates_2026-10-23.csv` | whenever the file lands | a signal arrived |
Cron: when the clock is the contract
A cron expression has 5 fields: minute, hour, day of month, month and day of week, so 0 6 * * 1-5 means 06:00, Monday to Friday. Cron fits when the data is complete by a known time and the consumer reads at a known time. A finance report opened at 09:00, built from a ledger that closes at midnight, belongs on cron at 02:00, leaving hours of slack for a slow night.
Cron trusts the clock and never looks at the data. If the ledger closes late, the 02:00 run reads a half-written day and still finishes green. Cron also inherits the calendar's oddities. On 8 March 2026, New York clocks jumped from 02:00 to 03:00, so a job set for 02:30 local time had no 02:30 to fire at; on 1 November 2026, 01:30 happens twice. Depending on the scheduler, such a run is skipped, shifted or doubled. Writing every schedule in UTC removes both cases.
Interval: an even rhythm, anchored to a start
An interval schedule says "every 15 minutes" and counts from an anchor, usually the moment the DAG was switched on. Anchored at 10:07, it starts runs at 10:07, 10:22 and 10:37, where */15 would start them at 10:15, 10:30 and 10:45. Both give equal windows. Pick interval when the gap matters and the clock positions do not, or when cron cannot express the gap: every 90 minutes takes 2 cron lines, because the minute alternates between :00 and :30.
The gap also caps freshness. A row that lands a second after a run reads its source waits a full gap for the next run, then waits for that run to finish. With a 15-minute gap and a 5-minute load, the dashboard can show data up to 20 minutes old. Gap plus run time must stay within the consumer's age limit.
A dashboard must show orders no older than 20 minutes, and each load takes 6 minutes. What is the longest gap that meets the limit?
- Every 20 minutes
- Every 10 minutes
- Every 26 minutes
- Every 14 minutes
Event: when arrival time is unpredictable
Some data arrives on its own clock. A partner uploads a file sometime between 22:00 and 05:00. A cron at 05:30 is safe, but on a 22:00 night the consumer waits 7.5 hours for data that was ready. An event schedule starts the run when the file lands, so the data reaches the consumer within minutes and no run reads a missing file. The cost is predictability: run times move every night, and if the signal never comes, nothing runs and nothing fails. Event schedules need a separate check that notices silence, covered with sensors and external triggers below.
| Data arrives | Consumer expects data | Choose |
|---|---|---|
| complete by a known time | ready at a fixed reading time | cron, after the usual arrival plus slack |
| continuously, all day | no older than a stated age | interval or cron, with gap plus run time within that age |
| at unpredictable times | soon after it lands | event trigger, plus an alert for silence |
| at unpredictable times | ready at a fixed reading time | cron after the latest arrival, or event plus deadline alert |
A bank sends a settlement file. Over the last 30 days it landed between 01:10 and 04:50 UTC. Treasury wants each file's totals within 30 minutes of landing. Which schedule fits?
- Cron at 05:00 UTC, after every arrival in the last 30 days
- Event trigger on the file landing, plus an alert if none by 06:00
- Interval every 30 minutes, loading any new file present when it fires
- Cron at 01:10 UTC, the earliest arrival in the last 30 days
About this lesson
This section is part of the Orchestration: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.