Sensors and External Triggers
Asset triggers need the write to happen where the orchestrator can see it. Many inputs are produced elsewhere: a bank drops a file on a server, a vendor's API marks a report ready, a partner writes into your storage bucket. For those, the orchestrator has 2 ways to learn that data exists: it asks, with a sensor, or it is told, with an external trigger.
Where a sensor belongs
A sensor belongs at the boundary: the first task of the DAG that consumes the outside data, waiting on the exact object that run needs. For the 2026-10-23 run, that is landing/rates/2026-10-23.csv, a path built from the run's data interval. A sensor waiting for any file in landing/rates/ succeeds on yesterday's file, and a sensor in the middle of a DAG lets earlier tasks run before the input exists, then holds the run open for hours.
Timeouts decide what silence means
A sensor without a timeout waits forever, and a run that waits forever alerts nobody. Work the timeout back from the consumer's deadline. Treasury reads the rates report at 07:00 and the load plus report take 40 minutes, so the latest useful arrival is 06:20. A sensor starting at 22:00 gets a timeout of 8 hours 20 minutes and fails at 06:20, early enough for someone to call the bank. Check every few minutes, since each check interval adds at most that much delay and faster polling loads the partner's server for nothing the consumer sees.
Sensor execution modes and worker slots
A worker slot is a task attempt's place on a worker, so a cluster with 32 slots runs 32 attempts at once. A sensor's execution mode decides how long it holds a slot while it waits.
| Mode | Slot held while waiting | Each check is | Fits |
|---|---|---|---|
| Blocking (poke) | yes, for the whole wait | a loop inside a single attempt | waits of a few minutes with frequent checks |
| Reschedule | only during each check | a short attempt started on a timer | waits of hours, checked every few minutes |
| Deferred | none; an async process holds the wait | a callback from that process | hundreds of long waits at the same time |
Suppose 40 partner files each get a blocking sensor that starts at 22:00 and waits around 6 hours, on a cluster of 32 slots. By 22:01 the sensors hold every slot and the rest queue. When a file lands, its sensor frees a slot, and the transform for that file competes for it with the sensors still queued. The same sensors in reschedule mode, checking every 5 minutes, hold a slot for a few seconds per check and leave the cluster to real work.
- 8
- 72
External triggers: let the source call in
An external trigger reverses the direction, and it lets finance drop the 01:00 cron from fx_rates_daily. The storage bucket emits an event when landing/rates/2026-10-23.csv is written, and a small function receives it and calls the orchestrator's API to start fx_rates_daily, passing the file path in the run's configuration. Nothing polls, and the run starts seconds after the write. Events can also be lost or duplicated. A lost event means no run and no error, so back every external trigger with a deadline check that alerts if no run has started by the consumer's cutoff. A duplicated event means 2 runs for 1 file, and using the file name as the run id makes the orchestrator reject the second as a run that already exists.
- partner file drop (source, partner SFTP server)
- landing bucket (storage, object storage)
- object-created event (queue, storage notification)
- trigger function (custom, calls orchestrator API)
- fx_rates_daily (transform, orchestrator DAG); idempotency: run id = file name; alerting: no run by 06:20 UTC
- warehouse.fx_rates (storage, warehouse table)
- partner file drop -> landing bucket
- landing bucket -> object-created event
- object-created event -> trigger function
- trigger function -> fx_rates_daily
- landing bucket -> fx_rates_daily
- fx_rates_daily -> warehouse.fx_rates
fx_rates_daily starts from an external trigger and has no deadline check. One night the storage event is lost. What does the team see?
- The sensor in fx_rates_daily times out and pages the on-call engineer
- No run starts and nothing fails, so no alert reaches anyone
- The orchestrator retries the trigger until the event gets delivered
- A failed run appears in the UI with an error about the event
- Do
- Wait on the exact file or partition named for this run's interval
- Derive the timeout from the consumer's deadline
- Back every external trigger with a deadline alert
- Avoid
- Sense another DAG where an asset trigger fits
- Hold a worker slot through a wait of hours
- Poll a partner's server every few seconds
A vendor posts a report between 23:00 and 03:00. The consumer reads at 08:00, and the downstream load takes 1 hour. You run 200 such sensors each night on a 64-slot cluster. Which sensor setup fits?
- Blocking mode, timeout at 07:00, checking every 10 minutes
- Reschedule mode, no timeout, checking every 10 seconds
- Deferred mode, timeout at 07:00, checking every 10 minutes
- Deferred mode, timeout at 08:00, checking every 10 minutes
About this lesson
This section is part of the Orchestration: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.