Cross-DAG Dependencies

Once 2 teams own 2 DAGs, one DAG's output becomes another's input, and the link between them sits outside both graphs. fx_rates_daily belongs to the finance data team and loads each day's exchange rates into warehouse.fx_rates. daily_revenue belongs to analytics and reads that table. The orchestrator links them only if you build the link, with a time offset, a sensor or an asset trigger.

Time offset: a dependency nobody wrote down

Schedule fx_rates_daily at 01:00 and daily_revenue at 03:00, because the rates load has finished by 02:40 on every night anyone checked. Neither DAG records the dependency. It costs nothing to build and fails silently: on a night when the load takes 2 hours 25 minutes, daily_revenue reads the previous day's rates at 03:00, succeeds, and publishes. Offsets also decay, since the upstream grows slower month by month and nobody downstream hears when it crosses 03:00.

Sensor: wait for the upstream task

A sensor is a task whose only job is to check a condition on a timer; it succeeds once the condition holds. Put one first in daily_revenue, waiting for load_rates in fx_rates_daily to succeed for the matching data interval. A slow night now delays revenue instead of corrupting it, and a missed night fails the sensor at its timeout. The cost is coupling. The sensor names another team's DAG id and task id, and it finds the matching upstream run from a fixed time difference between the 2 schedules. If finance renames load_rates, or moves the DAG from 01:00 to 01:30, the sensor waits for a task instance that will never exist.

Asset trigger: depend on the table itself

An asset trigger moves the contract from the DAG to the data. The load_rates task declares that it updates the asset warehouse.fx_rates, and when that task instance succeeds, the orchestrator records an update event. daily_revenue names the asset as its schedule, so each update event creates a run. Downstream now knows a table name and nothing about who writes it or when: finance can rename tasks, split the DAG or move it to 01:30 without telling analytics. Revenue also starts seconds after the rates land, on slow nights and fast ones.

- dag_id: fx_rates_daily
  schedule: "0 1 * * *"
  tasks:
    - id: load_rates
      produces: [warehouse.fx_rates]
- dag_id: daily_revenue
  schedule:
    on_assets: [warehouse.fx_rates]
  tasks:
    - id: extract_bookings
    - id: convert_currency
      depends_on: [extract_bookings]

Asset triggers have 2 limits. An update event says the asset changed and says nothing about which slice of data changed, so a downstream that cares must check the content, which the last section of this lesson builds. And the event exists only if the write happened inside the same orchestrator, or the outside writer reports it through the orchestrator's API.

Which failure can a time offset produce that a sensor and an asset trigger both prevent?

• The downstream waits until a timeout and then fails
• The downstream breaks when the upstream renames a task
• The downstream starts once for every upstream update
• The downstream succeeds on stale input with no error

orders_load runs hourly in your orchestrator and writes warehouse.orders. Its owners reorganize its tasks every quarter. Your DAG must start within minutes of each load. Which dependency fits?

• An asset trigger on the warehouse.orders table
• A sensor on the final task of orders_load
• A schedule 20 minutes after every hour
• A sensor on the first task of orders_load

About this lesson

This section is part of the Orchestration: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.