Orchestration: Intermediate

A cron expression has 5 fields: minute, hour, day of month, month and day of week, so 0 6 * * 1-5 means 06:00, Monday to Friday. Cron fits when the data is complete by a known time and the consumer reads at a known time. A finance report opened at 09:00, built from a ledger that closes at midnight, belongs on cron at 02:00, leaving hours of slack for a slow night.

An interval schedule says "every 15 minutes" and counts from an anchor, usually the moment the DAG was switched on. Anchored at 10:07, it starts runs at 10:07, 10:22 and 10:37, where */15 would start them at 10:15, 10:30 and 10:45. Both give equal windows. Pick interval when the gap matters and the clock positions do not, or when cron cannot express the gap: every 90 minutes takes 2 cron lines, because the minute alternates between :00 and :30.

When something breaks at 04:00, people use 5 words loosely, and the right fix depends on which one you mean. Take a DAG named daily_revenue with 3 tasks in a line: extract_bookings, then convert_currency, then publish_report.

Success and failure live on task_instance rows, so every fix below acts on runs and task instances.

Say convert_currency fails in the 2026-10-23 run with a retry count of 2. The orchestrator leaves the rest of the run alone: it moves that task instance to up for retry, waits the configured delay, and starts attempt 2. extract_bookings keeps its success and doesn't execute again, and publish_report keeps waiting. A retry count of 2 allows 3 attempts. Only when attempt 3 fails does the task instance become failed, and only then does the run fail.

The common mistake is clearing too little. Suppose extract_bookings for 2026-10-23 succeeded while its source was still filling in. You wait for the source to finish and clear only extract_bookings. It reruns and succeeds, while convert_currency and publish_report keep their earlier success, so the report still shows partial numbers. Once a task's output changes, every task that read it must run again in the same run, which is what clearing with downstream does.

Once 2 teams own 2 DAGs, one DAG's output becomes another's input, and the link between them sits outside both graphs. fx_rates_daily belongs to the finance data team and loads each day's exchange rates into warehouse.fx_rates. daily_revenue belongs to analytics and reads that table. The orchestrator links them only if you build the link, with a time offset, a sensor or an asset trigger.

Schedule fx_rates_daily at 01:00 and daily_revenue at 03:00, because the rates load has finished by 02:40 on every night anyone checked. Neither DAG records the dependency. It costs nothing to build and fails silently: on a night when the load takes 2 hours 25 minutes, daily_revenue reads the previous day's rates at 03:00, succeeds, and publishes. Offsets also decay, since the upstream grows slower month by month and nobody downstream hears when it crosses 03:00.

A sensor is a task whose only job is to check a condition on a timer; it succeeds once the condition holds. Put one first in daily_revenue, waiting for load_rates in fx_rates_daily to succeed for the matching data interval. A slow night now delays revenue instead of corrupting it, and a missed night fails the sensor at its timeout. The cost is coupling. The sensor names another team's DAG id and task id, and it finds the matching upstream run from a fixed time difference between the 2 schedules. If finance renames load_rates, or moves the DAG from 01:00 to 01:30, the sensor waits for a task instance that will never exist.

An asset trigger moves the contract from the DAG to the data. The load_rates task declares that it updates the asset warehouse.fx_rates, and when that task instance succeeds, the orchestrator records an update event. daily_revenue names the asset as its schedule, so each update event creates a run. Downstream now knows a table name and nothing about who writes it or when: finance can rename tasks, split the DAG or move it to 01:30 without telling analytics. Revenue also starts seconds after the rates land, on slow nights and fast ones.

A sensor belongs at the boundary: the first task of the DAG that consumes the outside data, waiting on the exact object that run needs. For the 2026-10-23 run, that is landing/rates/2026-10-23.csv, a path built from the run's data interval. A sensor waiting for any file in landing/rates/ succeeds on yesterday's file, and a sensor in the middle of a DAG lets earlier tasks run before the input exists, then holds the run open for hours.

An external trigger reverses the direction, and it lets finance drop the 01:00 cron from fx_rates_daily. The storage bucket emits an event when landing/rates/2026-10-23.csv is written, and a small function receives it and calls the orchestrator's API to start fx_rates_daily, passing the file path in the run's configuration. Nothing polls, and the run starts seconds after the write. Events can also be lost or duplicated. A lost event means no run and no error, so back every external trigger with a deadline check that alerts if no run has started by the consumer's cutoff. A duplicated event means 2 runs for 1 file, and using the file name as the run id makes the orchestrator reject the second as a run that already exists.

A first draft schedules revenue_by_currency at :30 past every hour, reasoning that orders finish by :06, payments by :15 and rates by 06:20. That stacks 3 time offsets, each failing silently: a payments file 40 minutes late means the 14:30 run builds 13:00 to 14:00 without payments and publishes it as final. Sensors fit badly too: the event-triggered rates DAG has no schedule from which to compute the run to wait for, and mapping each hourly run to 4 orders runs works but hard-codes both teams' cadences.

In the design that holds up, every upstream DAG ends with a task that records how far its table is loaded, as a row in pipeline_freshness, and that task declares the table as an asset. Then revenue_by_currency runs whenever any of the 3 assets updates, and its first task asks the freshness table a single question: up to which hour are all 3 inputs complete?

The run then builds every hour that ends by build_through and has not been built yet, and it writes build_through beside the output so the dashboard can label its numbers final through 14:00. If inputs_reporting is below 3, an upstream has never reported, and the run fails loudly. The slowest input sets the line, whichever input that happens to be.

When a payments file is 2 hours late, orders updates keep starting runs that find build_through stuck at 14:00, so the dashboard keeps saying final through 14:00 rather than publishing the hour ending 15:00 with no payments. The alert pages the owner of slowest_input, with a threshold per input: orders or payments at 90 minutes behind the clock, rates only if still slowest after 06:20, since until the bank's file lands they hold build_through at midnight every night.