Reading a Pipeline Left to Right
A pipeline diagram is read the way data moves through it: left to right, source to consumer, along the arrows. Each arrow means the box at its head reads what the box at its tail wrote, and nothing more. Boxes with no arrow in are sources, and boxes with no arrow out are consumers. Everything between is a transform or storage, and the question that tells those 2 apart is whether the box still holds data after the run ends.
- orders (source, Postgres)
- extract_orders (transform, Python)
- shop-raw (storage, S3 Parquet)
- clean_and_sum (transform, dbt)
- daily_category_revenue (storage, Snowflake)
- Revenue dashboard (consumer, Looker)
- orders -> extract_orders
- extract_orders -> shop-raw
- shop-raw -> clean_and_sum
- clean_and_sum -> daily_category_revenue
- daily_category_revenue -> Revenue dashboard
- Source: orders
- Raw: extract_orders, shop-raw
- Serving: clean_and_sum, daily_category_revenue, Revenue dashboard
This drawing has 6 boxes where the previous section's had 4, and the 4 roles still account for every one of them. The extraction is now its own transform, because copying rows out of Postgres and writing them as Parquet changes their format even though it changes no values. The S3 bucket is storage sitting in the middle of the flow: it has an arrow in and an arrow out, and the Parquet files are still there a month later. A box with both arrows that keeps its data is storage a later step reads; a box with both arrows that keeps nothing is a transform.
3 questions to ask of every box
- What does it read? Follow the arrows in. A source reads nothing from the pipeline.
- What does it write? Follow the arrows out. A consumer writes nothing back into the pipeline.
- Does it keep what it wrote? Storage does. A transform hands its output on and forgets it.
Asked of every box in turn, those 3 questions name its role without reading a single label. Labels help afterwards: a Postgres box on the far left is almost certainly an operational source, and a Looker box on the far right a dashboard. Labels in the middle are where readers get lost, because a Snowflake box can be storage that a dbt transform reads, and a dbt box can appear twice in one diagram doing 2 different jobs.
One order, traced along the arrows
| Time on 14 and 15 March | Box | What happens to order 88213 |
|---|---|---|
| 14:32, 14 Mar | orders | Written as 1 row: amount '24.99' stored as text, time in Europe/London |
| 02:03, 15 Mar | extract_orders | Read with the day's other 6,213 rows and written unchanged to Parquet |
| 02:03, 15 Mar | shop-raw | Sits in s3://shop-raw/orders/date=2024-03-14/ with its original values |
| 02:10, 15 Mar | clean_and_sum | Amount cast to 24.99, time shifted to UTC, summed into the kettles row for the day |
| 02:22, 15 Mar | daily_category_revenue | Present only as part of 1 category row: kettles, 1,088 orders, 41,208.40 |
| 08:00, 15 Mar | Revenue dashboard | Chart reads the kettles row among 240 rows for the month |
The trace shows what left to right costs the individual record. Order 88213 is a row of its own in Postgres and in S3, and after the transform it exists only inside a category total. Anyone asking the dashboard about that 1 order has to go back along the arrows to a box that still holds it, which is why the diagram keeps the raw copy in S3.
A diagram shows this flow: web_clicks (Kafka) to parse_events (Flink) to events_raw (S3) to sessionize (Spark) to sessions (BigQuery) to retention_dashboard (Looker). Which boxes still hold data after a run finishes?
- parse_events and sessionize, because they do the work
- events_raw and sessions, the 2 storage boxes in the middle and near the end
- Only sessions, because it is the last box before the dashboard
- Every box, since data passes through all of them
Direction is the other thing readers reverse. A dashboard box on the right receives no push from the warehouse; it runs a query, and the arrow records that read. The same holds for every arrow in the diagram.
An arrow runs from daily_category_revenue (Snowflake) to Revenue dashboard (Looker). What does that arrow say?
- Looker runs a query that reads the table
- Snowflake pushes new rows to Looker whenever they land
- Looker writes its chart results back into the table
- Data can move in either direction depending on the day
About this lesson
This section is part of the Pipeline Anatomy: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.