When to Split, When to Merge
Concepts covered: DAG Orchestration
A DAG, a directed acyclic graph, is one scheduled graph of steps that runs as a unit: the arrows point forward and no step waits on itself. When a step fails, every step downstream of it is skipped, and steps on other branches still run. So a fragile step upstream of an urgent one sits on its critical path. Put too much into 1 DAG and a step nobody cares about ends up upstream of the step everyone cares about. Split into too many and every boundary becomes a handoff that nobody monitors. 3 signals decide where the edges go. Ownership and cadence are the first 2; the third is what it costs to operate the boundary.
The 3 splitting signals
- Split when
- 2 teams own steps in the same graph, and 1 team's bug pages the other
- Steps need different cadences: an hourly step inside a nightly DAG runs 24 times or once, never right
- A failure in 1 consumer's steps should not block another consumer: 2 consumers of 1 source each get their own graph
- A rerun of 1 half rescans the other half for no reason
- Merge when
- A step exists only to feed the next, and no other consumer reads its output
- Both sides are owned by the same team and run at the same cadence
- The boundary would need its own contract and a ready-time check with an alert on it, and nobody would own them
- Each half would be fewer than 3 steps and the halves would always run together
Worked example: orders and inventory
An online store runs 1 nightly DAG: pull orders, pull inventory, clean both together, compute daily revenue, compute low-stock alerts. The low-stock alerts are wanted every hour, so the team schedules the whole DAG hourly. Now daily revenue is recomputed 24 times a day, the orders database is pulled 24 times, and when an orders region file is missing at 14:00, the shared cleaning step fails and no low-stock alert fires until the file appears. Cadence is the signal: 2 cadences in 1 DAG, and the slow half is on the fast half's critical path.
- orders_db (source, transactional database)
- ingest_orders (transform, scheduler)
- clean_orders (storage, curated table)
- daily_revenue (consumer, finance report); freshness: by 07:00 UTC
- inventory_db (source, transactional database)
- ingest_inventory (transform, scheduler)
- low_stock_alerts (consumer, alerting); freshness: every hour
- orders_db -> ingest_orders
- ingest_orders -> clean_orders
- clean_orders -> daily_revenue
- inventory_db -> ingest_inventory
- ingest_inventory -> low_stock_alerts
- Nightly DAG: orders_db, ingest_orders, clean_orders, daily_revenue
- Hourly DAG: inventory_db, ingest_inventory, low_stock_alerts
After the split, the orders database is pulled once a night and the inventory database once an hour. The boundary costs almost nothing because the 2 graphs share no step, so there is no handoff to monitor. The decision gets harder when they do share one.
When they share a step
Suppose low-stock alerts need clean_orders too, to know what sold in the last hour. Now that table has 2 consumers at 2 cadences. The split still holds, but the shared step either moves into the hourly graph, which then pulls orders hourly, or becomes a product of its own with a ready-time check that both graphs read. The rule from the product framing applies: a step read by another DAG is a product and gets a contract, even a 3-line one. Merging the 2 DAGs back together to avoid that contract trades 1 monitored handoff for the 24-times-a-day revenue problem, which is the worse deal.
A single DAG owned by the marketing team has 9 steps. Step 7, finance_close, is owned by finance and is the only step finance uses. It runs nightly like the rest and fails about once a week on a finance-side input file. Should the DAG split, and on which signal?
- Split on ownership: a finance file failure pages marketing, and marketing cannot fix it
- Keep it merged: the cadences match, so a split would add a boundary with no benefit
- Split on cadence: a weekly failure means finance_close wants a weekly schedule
- Split on cost: 9 steps is past the size at which a DAG should always be divided
About this lesson
This section is part of the Pipeline Anatomy: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.