Redesigning a Tangled Graph
Concepts covered: DAG Orchestration
Here is a graph as it looks after 3 years of growth: 14 steps owned by 3 teams, all on 1 nightly schedule. Each step was added by whoever needed it at the time, reading from whichever step already had the data. Read it the way the intermediate lesson taught, left to right, then look for arrows that point the wrong way.
- orders_db (source, transactional database)
- web_events (source, event log)
- crm_db (source, CRM)
- ingest_orders (transform, scheduler)
- ingest_events (transform, scheduler)
- ingest_crm (transform, scheduler)
- orders_clean (storage, platform team)
- sessions (storage, platform team)
- customer_360 (storage, platform team)
- attribution (transform, marketing team)
- daily_revenue (transform, finance team)
- finance_close (transform, finance team)
- refund_adjustments (transform, finance team)
- exec_dashboard (consumer, dashboard)
- orders_db -> ingest_orders
- web_events -> ingest_events
- crm_db -> ingest_crm
- ingest_orders -> orders_clean
- ingest_events -> sessions
- ingest_crm -> customer_360
- orders_clean -> customer_360
- sessions -> customer_360
- sessions -> attribution
- orders_clean -> attribution
- attribution -> daily_revenue
- orders_clean -> daily_revenue
- daily_revenue -> finance_close
- customer_360 -> finance_close
- finance_close -> refund_adjustments
- refund_adjustments -> exec_dashboard
- attribution -> exec_dashboard
- customer_360 -> exec_dashboard
- daily_revenue -> exec_dashboard
Diagnosis
Apply the 3 framings in order. Products first: accounting reads the close, marketing reads its attribution report, and executives read the dashboard. Behind them sit 3 products with 3 contracts: finance_close and attribution, plus the curated layer that both DAGs and the dashboard read, which holds orders_clean and sessions along with customers. Undercurrents second: walk the failures row and the critical path shrinks to the steps finance needs. Split third: 3 owners and 2 cadences, so 3 DAGs around a curated layer the platform team owns.
The redesign
- orders_db (source, transactional database)
- web_events (source, event log)
- crm_db (source, CRM)
- ingest_orders (transform, scheduler)
- ingest_events (transform, scheduler)
- ingest_crm (transform, scheduler)
- orders_clean (storage, curated table); freshness: by 03:00 UTC
- sessions (storage, curated table); freshness: by 03:00 UTC
- customers (storage, curated table); freshness: by 03:00 UTC
- daily_revenue (transform, finance DAG)
- refund_adjustments (transform, finance DAG)
- finance_close (storage, finance product); freshness: by 06:00 UTC
- attribution (storage, marketing product); freshness: Monday 08:00 UTC
- exec_dashboard (consumer, dashboard)
- orders_db -> ingest_orders
- web_events -> ingest_events
- crm_db -> ingest_crm
- ingest_orders -> orders_clean
- ingest_events -> sessions
- ingest_crm -> customers
- orders_clean -> customers
- sessions -> customers
- orders_clean -> daily_revenue
- orders_clean -> refund_adjustments
- daily_revenue -> finance_close
- refund_adjustments -> finance_close
- customers -> finance_close
- sessions -> attribution
- orders_clean -> attribution
- finance_close -> exec_dashboard
- attribution -> exec_dashboard
- customers -> exec_dashboard
- Sources: orders_db, web_events, crm_db
- Curated, platform DAG: ingest_orders, ingest_events, ingest_crm, orders_clean, sessions, customers
- Finance DAG: daily_revenue, refund_adjustments, finance_close
- Marketing DAG: attribution
- Serving: exec_dashboard
| Product | Owner | Consumers | Ready by | Change policy |
|---|---|---|---|---|
| curated layer: `orders_clean`, `sessions`, `customers` | platform | finance DAG, marketing DAG, dashboard | 03:00 UTC | additive only, 30 days notice |
| `finance_close` | finance | dashboard, accounting export | 06:00 UTC | column set frozen for the fiscal year |
| `attribution` | marketing | dashboard | Monday 08:00 UTC | model version in a column, old column kept 2 cycles |
Check the result against the undercurrents. Failures: a marketing model failure now delays 1 weekly report, and finance closes on time. Reruns: a curated rerun clears the ready marker first, then writes each day aside and swaps it in at once, so nobody reads a half-written day at 03:00 or at 10:00. Lateness: late refunds reopen refund_adjustments and the close, and the finance contract says so. Change: customer_360 was renamed customers through the deprecation path, with both names served for 30 days. Visibility: each product records its ready time, so the 09:00 question has a dashboard answer. Cost: the attribution model runs 1 night in 7 instead of 7.
Capstone
The marketing team, 6 months after the redesign, wants attribution to run nightly and asks to move it into the finance DAG so it can reuse the finance schedule. Using the 3 splitting signals, what is the right response?
- Agree: the cadences now match, which was the signal that split them in the first place
- Decline: ownership still differs, so attribution gets its own nightly schedule reading the same curated layer
- Agree, as long as marketing writes a contract for attribution before the move
- Decline: the finance DAG already holds 3 steps and cannot take a 4th
Retiring a single table needs a finer list than the products table gives: every reader of that one table, including steps inside the owning DAG, the readers teams most often forget.
The platform team wants to retire sessions and replace it with sessions_v2, which has a different grain. Which part of the redesign makes this possible without a surprise, and what does the team do first?
- The curated layer's change policy; the team drops sessions and announces sessions_v2 the same day
- The finance DAG's 06:00 ready time; the team ships sessions_v2 before 06:00 so nothing downstream notices
- The 30-day notice in the contract; the team ships sessions_v2 beside sessions and works through every reader of sessions
- The weekly cadence of attribution; the team swaps the table on a Tuesday when the model is not running
About this lesson
This section is part of the Pipeline Anatomy: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.