Two DAGs are glued by a time offset: the orders DAG schedules at 2am and usually finishes by 2:45am,

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

Two DAGs are glued by a time offset: the orders DAG schedules at 2am and usually finishes by 2:45am, and the marketing DAG schedules at 3am hoping the upstream is done. When the orders DAG runs slow and finishes at 3:15am, the marketing DAG already started at 3am and read stale state. Apply the cross-DAG framing this section just taught and replace the time offset with an asset trigger. The asset is structurally a lakehouse-format table (Iceberg, Delta, or Hudi) that the orders DAG writes and the marketing DAG reads; the lakehouse format gives the table snapshot semantics that asset-aware orchestrators trigger on. Replace the existing plain Snowflake mart with a lakehouse-format table; add a second orchestrator (or use the same orchestrator's asset-trigger feature) so the marketing DAG fires when the asset is fresh.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.