Pipeline Anatomy: Advanced
Name the consumer
| Consumer | What they read | What breaks them |
|---|---|---|
| Recommendations | sessions from the last 24 hours, by 06:00 | a table that lands at 09:00 misses the morning model refresh |
| Royalty reporting | every session in a calendar month | a day with 2% of rows missing understates payouts |
| Yearly recap campaign | 12 months of sessions, read once in December | a renamed `ms_played` column 3 days before launch |
Without this list, every design decision is a guess. Should the pipeline ship a partial day at 06:00 or a complete day at 09:00? Recommendations wants partial and royalties wants complete. The answer is 2 outputs: a listening_sessions_early snapshot at 06:00 with recommendations as its only consumer, and the final table at 09:00 for everyone else.
Write the contract
Each line of the contract maps to a check the pipeline runs on itself. The ready_by and completeness lines become the query below, run at 09:00. It fails if the curated table holds under 99.5% of the sessions that landed in raw.listening_sessions for yesterday, or if yesterday's latest session started before 23:00, a sign the day was cut off. The columns line becomes a check on names and types.
Define the deprecation path
Every table is eventually replaced. The product framing plans that replacement on day 1, because the consumer list is the only way to know who the replacement will hurt. The path has 4 steps, and the contract's consumers field drives the 3rd.
A team runs a nightly job that writes orders_daily. It has run for 3 years, 6 dashboards query it, and nobody on the team can say which 6. Which single change moves it furthest toward being a product?
The 6 undercurrents
- Rerunsa step runs twiceA nightly load fails at 02:10 and someone restarts it at 08:00. If the first attempt had already written 400,000 rows, does the table now hold 800,000?
- Failuresa step stops partwayThe transform step dies after writing 3 of 12 regions. Does the serving step wait, run on the 3, or run on yesterday's 12?
- Changea source changes shapeThe orders service renames amount to amount_cents. Which step is the first to notice, and is that step a check or a dashboard?
- Latenessa row arrives after its day closedA session that ended at 23:58 on Tuesday reaches the pipeline at 00:40 on Wednesday, after Tuesday's totals were served. Does Tuesday ever get corrected?
- Visibilitysomeone asks if the table is rightAt 09:00 an analyst asks whether yesterday is complete. Can anyone answer from a dashboard, or only by running queries by hand?
- Costthe bill arrivesThe pipeline scans 2 TB a night. Is that because consumers need 2 TB, or because a rerun also rescans 30 days it did not need to?
| Undercurrent | Source | Ingest | Transform | Serve |
|---|---|---|---|---|
| Reruns | can the source be read twice for the same window | does a 2nd pull for the same window replace the 1st | does rerunning a day overwrite that day's output | does a consumer see the half-written state mid-rerun |
| Failures | does the source return partial data or an error | is a partial pull discarded or kept | does a step that dies leave a marker the next step reads | does serving fall back to the last complete day |
| Change | does the source announce changes to its shape | are unknown columns kept, dropped, or rejected | does a renamed column fail the step or silently null the output | does the consumer's contract say which columns are stable |
| Lateness | how long after an event can the source still emit it | is the pull window wider than the lateness | are closed days reopened, and how many days back | does the consumer know a day's number may move |
| Visibility | does the source expose its own row counts | is the pulled row count recorded per run | is output row count compared with input | is each day's ready time recorded where consumers look |
| Cost | does each pull hit a production database | is the pull incremental or full | does a rerun touch 1 day or 30 | does each consumer's query read only the partitions it needs |
Walking a design through the map
- ▸Lateness forces reruns: a day that reopens is a day that is written twice
- ▸Reruns force cost: every reopened day is rescanned, so the rerun window is a bill
- ▸Failures force visibility: a step that stops partway is safe only if the next step can see it stopped
- ▸Change forces contracts: a renamed column is harmless to a consumer whose contract never promised it
A pipeline serves daily revenue. On Thursday a consumer notices that Monday's revenue figure changed by 0.4% overnight, with no alert and no note anywhere. Which cell of the map holds the gap?
The 3 splitting signals
- 2 teams own steps in the same graph, and 1 team's bug pages the other
- Steps need different cadences: an hourly step inside a nightly DAG runs 24 times or once, never right
- A failure in 1 consumer's steps should not block another consumer: 2 consumers of 1 source each get their own graph
- A rerun of 1 half rescans the other half for no reason
- A step exists only to feed the next, and no other consumer reads its output
- Both sides are owned by the same team and run at the same cadence
- The boundary would need its own contract and a ready-time check with an alert on it, and nobody would own them
- Each half would be fewer than 3 steps and the halves would always run together
Worked example: orders and inventory
After the split: the nightly DAG and the hourly DAG share no steps, so a missing region file at 14:00 delays revenue and nothing else.
When they share a step
Suppose low-stock alerts need clean_orders too, to know what sold in the last hour. Now that table has 2 consumers at 2 cadences. The split still holds, but the shared step either moves into the hourly graph, which then pulls orders hourly, or becomes a product of its own with a ready-time check that both graphs read. The rule from the product framing applies: a step read by another DAG is a product and gets a contract, even a 3-line one. Merging the 2 DAGs back together to avoid that contract trades 1 monitored handoff for the 24-times-a-day revenue problem, which is the worse deal.
A single DAG owned by the marketing team has 9 steps. Step 7, finance_close, is owned by finance and is the only step finance uses. It runs nightly like the rest and fails about once a week on a finance-side input file. Should the DAG split, and on which signal?
The 3 weights
- Differentiationdoes building it make the product better for the consumerA connector that pulls a payment provider's API is identical whoever writes it; the consumer never sees it. A transform that encodes how your company defines an active user is yours alone, and buying it means buying someone else's definition.
- Total cost of ownershipbuild cost plus every year of running itThe build is the cheap part. A connector takes 6 engineer-weeks to write and 1 engineer-day a month forever to keep working as the source changes, which is 2.5 more engineer-weeks a year, every year.
- Strategic riskwhat happens if the choice has to be undoneA bought storage layer holds your data in its format; leaving means a migration measured in months. A bought connector holds nothing; leaving means rewriting 1 pull. Risk is highest where data and logic accumulate.
| Layer | Default | Exception |
|---|---|---|
| Ingest connectors | buy | the source is internal and proprietary, so no vendor sells a connector for it |
| Storage | buy | regulation requires the bytes to stay on hardware you control |
| Transform logic | build | the logic is a commodity, such as currency conversion from a published rate table |
| Transform engine | buy | a query shape the managed engine cannot finish inside the contract's ready-by time |
| Scheduler | buy | schedules depend on signals from internal systems the managed scheduler cannot see |
| Serving | buy | the consumer is a product feature with latency needs no dashboard tool meets |
A worked total cost
- Decide per layer; buying storage and building transforms is the common shape
- Count maintenance in the build cost, in engineer-days per month, forever
- Ask which layer your data and logic would be stuck in if the vendor vanished
- Build a connector to a public API because it looks like a 2-week job
- Buy transform logic; the active-user definition is the product
- Compare year-1 costs only; the crossover is usually in year 2 or 3
A team is choosing where the transform that defines a 'paying customer' lives. Finance and sales disagree on that definition, and support has a third version. A vendor sells a package that computes paying customers from billing data. Which weight decides this, and which way?
14 steps on one nightly schedule. Marketing and finance own some of them and the platform team owns the rest. The arrow from attribution into daily_revenue is the one to stare at.
Diagnosis
- ▸
daily_revenue, a finance figure, readsattribution, a marketing model: a change to the model moves the revenue number - ▸
exec_dashboardreads 4 steps across 3 teams, so any team's failure blanks the dashboard - ▸
customer_360has 3 consumers and no contract; it is a product nobody declared - ▸Because there is one schedule, the attribution model reruns every night even though it is wanted weekly, and it sits on the critical path to finance
- ▸
refund_adjustmentsruns afterfinance_close, so a refund posted at 23:00 reaches the dashboard but never the close
Apply the 3 framings in order. Products first: accounting reads the close, marketing reads its attribution report, and executives read the dashboard. Behind them sit 3 products with 3 contracts: finance_close and attribution, plus the curated layer that both DAGs and the dashboard read, which holds orders_clean and sessions along with customers. Undercurrents second: walk the failures row and the critical path shrinks to the steps finance needs. Split third: 3 owners and 2 cadences, so 3 DAGs around a curated layer the platform team owns.
The redesign
The 3 DAGs read 1 curated layer. Revenue no longer depends on the marketing model, refunds land before the close, and the dashboard reads 3 products instead of 4 internal steps.
| Product | Owner | Consumers | Ready by | Change policy |
|---|---|---|---|---|
| curated layer: `orders_clean`, `sessions`, `customers` | platform | finance DAG, marketing DAG, dashboard | 03:00 UTC | additive only, 30 days notice |
| `finance_close` | finance | dashboard, accounting export | 06:00 UTC | column set frozen for the fiscal year |
| `attribution` | marketing | dashboard | Monday 08:00 UTC | model version in a column, old column kept 2 cycles |
Check the result against the undercurrents. Failures: a marketing model failure now delays 1 weekly report, and finance closes on time. Reruns: a curated rerun clears the ready marker first, then writes each day aside and swaps it in at once, so nobody reads a half-written day at 03:00 or at 10:00. Lateness: late refunds reopen refund_adjustments and the close, and the finance contract says so. Change: customer_360 was renamed customers through the deprecation path, with both names served for 30 days. Visibility: each product records its ready time, so the 09:00 question has a dashboard answer. Cost: the attribution model runs 1 night in 7 instead of 7.
Capstone
The marketing team, 6 months after the redesign, wants attribution to run nightly and asks to move it into the finance DAG so it can reuse the finance schedule. Using the 3 splitting signals, what is the right response?
The platform team wants to retire sessions and replace it with sessions_v2, which has a different grain. Which part of the redesign makes this possible without a surprise, and what does the team do first?
> A food delivery company has 1 nightly graph that feeds driver payouts, the restaurant partner dashboard, and the promotions model, which the growth team owns and needs refreshed only weekly. Payouts are late twice a month because the promotions model fails on a feature table it builds from the same deliveries data, and nobody on the payments team can say which dashboards read the deliveries table they maintain.
A pipeline is a product: name its consumer, write its contract, plan its retirement, then untangle the rest.
- Category
- Pipeline Architecture
- Difficulty
- advanced
- Duration
- 35 minutes
- Challenges
- 12 hands-on challenges
Topics covered: Pipelines as Products, The Cross-Cutting Undercurrents, When to Split, When to Merge, Build vs Buy at Each Layer, Redesigning a Tangled Graph
Lesson Sections
- Pipelines as Products (concepts: Data Quality & Validation)
The intermediate lesson drew pipelines as boxes and arrows: sources on the left, a curated layer in the middle, consumers on the right. Those diagrams answer where data moves. A product framing asks a second question of every box on the right edge: who reads this, and what did we promise them? A pipeline is a product when it has a named consumer and a written contract, and when someone has planned how it will be retired. A pipeline with none of those is a script that happens to run on a schedule
- The Cross-Cutting Undercurrents (concepts: DAG Orchestration)
The beginner lesson had 4 roles: source and consumer at the ends, with transform and storage between them. The undercurrent map regroups them into 4 columns that read left to right. Source stays as it was. Ingest is the first transform, the one that pulls data out of the source. Transform covers the remaining steps and the storage they write. Serve is the stored output as consumers read it. The undercurrents run the other way. Each is a concern no single box owns, so each shows up in all 4 colum
- When to Split, When to Merge (concepts: DAG Orchestration)
A DAG, a directed acyclic graph, is one scheduled graph of steps that runs as a unit: the arrows point forward and no step waits on itself. When a step fails, every step downstream of it is skipped, and steps on other branches still run. So a fragile step upstream of an urgent one sits on its critical path. Put too much into 1 DAG and a step nobody cares about ends up upstream of the step everyone cares about. Split into too many and every boundary becomes a handoff that nobody monitors. 3 signa
- Build vs Buy at Each Layer (concepts: ELT vs ETL)
Every layer of a pipeline can be built in-house or bought as a managed service: the connectors that pull from sources, the storage the curated layer sits in, the transform logic that encodes your definitions, the engine that runs that logic, the scheduler that runs the DAGs, and the tools that serve the output. The decision is made per layer, and the same team reasonably builds 1 of the 6 and buys the other 5. Each decision comes down to 3 weights. The 3 weights A worked total cost The listening
- Redesigning a Tangled Graph (concepts: DAG Orchestration)
Here is a graph as it looks after 3 years of growth: 14 steps owned by 3 teams, all on 1 nightly schedule. Each step was added by whoever needed it at the time, reading from whichever step already had the data. Read it the way the intermediate lesson taught, left to right, then look for arrows that point the wrong way. Diagnosis The redesign Capstone Retiring a single table needs a finer list than the products table gives: every reader of that one table, including steps inside the owning DAG, th