The Cross-Cutting Undercurrents
Concepts covered: DAG Orchestration
The beginner lesson had 4 roles: source and consumer at the ends, with transform and storage between them. The undercurrent map regroups them into 4 columns that read left to right. Source stays as it was. Ingest is the first transform, the one that pulls data out of the source. Transform covers the remaining steps and the storage they write. Serve is the stored output as consumers read it. The undercurrents run the other way. Each is a concern no single box owns, so each shows up in all 4 columns, and a design that handles it in 1 column and forgets the other 3 still fails.
The 6 undercurrents
- Reruns: A nightly load fails at 02:10 and someone restarts it at 08:00. If the first attempt had already written 400,000 rows, does the table now hold 800,000?
- Failures: The transform step dies after writing 3 of 12 regions. Does the serving step wait, run on the 3, or run on yesterday's 12?
- Change: The orders service renames amount to amount_cents. Which step is the first to notice, and is that step a check or a dashboard?
- Lateness: A session that ended at 23:58 on Tuesday reaches the pipeline at 00:40 on Wednesday, after Tuesday's totals were served. Does Tuesday ever get corrected?
- Visibility: At 09:00 an analyst asks whether yesterday is complete. Can anyone answer from a dashboard, or only by running queries by hand?
- Cost: The pipeline scans 2 TB a night. Is that because consumers need 2 TB, or because a rerun also rescans 30 days it did not need to?
Each undercurrent has a different answer in each role, which is why mapping them is worth a table. Read it one row at a time: the reruns row asks what the source does under a rerun, then what ingest does, then transform, then serve. A cell that nobody on the team can fill is a gap in the design, found before it is found in production.
Walking a design through the map
Take the listening sessions pipeline and walk the lateness row. Source: the mobile app buffers sessions while the phone is offline, so a session can arrive up to 3 days late. Ingest: the nightly pull reads yesterday only, so a 2-day-old session is never pulled. That cell already fails. Transform and serve never see the row, so nothing in their cells can recover it. The fix belongs in ingest: widen the pull to 3 days and rewrite those 3 days each night. That fix then changes the reruns row, because 3 days are rewritten and the write must replace rather than add, and the cost row, because 3 days are scanned instead of 1. One undercurrent pulled 2 others with it, which is the normal case.
A pipeline serves daily revenue. On Thursday a consumer notices that Monday's revenue figure changed by 0.4% overnight, with no alert and no note anywhere. Which cell of the map holds the gap?
- Transform, lateness: the step should refuse rows for a day that has already closed
- Ingest, reruns: the 2nd pull for Monday should have replaced the 1st instead of appending
- Source, change: the revenue source renamed a column and the pipeline did not notice
- Serve, lateness: the consumer was never told that a closed day's figure can move
About this lesson
This section is part of the Pipeline Anatomy: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.