The Four Roles in Any Pipeline
Every pipeline, from the shop's nightly copy to the systems behind a streaming service's recommendations, is built from the same 4 roles. Naming them turns any pipeline diagram into something readable, because every box in one is playing exactly 1 of them at that point in the flow.
Source
A source is a system that produces data as a side effect of running the business. The shop's Postgres orders table exists so the checkout can record a sale, and the fact that finance wants a chart from it played no part in its design. Other sources: a payments API such as Stripe, a clickstream from the website, log files from a server, a SaaS tool's export endpoint. Like the orders table, every source is shaped for the operational job rather than for analysis, and none can be asked to work harder without risk to the business function it exists for. Both facts shape everything downstream. A pipeline reads from a source and never writes back to one.
Transform
A transform is a step that reads data in 1 shape and writes it in another. The shop's transform converts amount strings to decimals, drops test orders, moves timestamps to UTC and sums revenue by category and day. Transforms range from a 20-line SQL query to a Spark job across 200 machines, and the technology varies more here than in any other role: Python scripts, dbt models, Spark, Flink. What identifies a transform is that its output differs in shape or content from its input, and that it holds no data of its own between runs. Stop a transform and nothing is lost except the run in progress.
Storage
Storage is a system that holds the transformed result so it can be read many times without repeating the work. The shop writes its 8 category rows a day into a Snowflake table, and for a month's chart the dashboard scans 240 rows instead of 186,000 orders. Storage in a pipeline is chosen for the reads that will hit it: a warehouse such as Snowflake or BigQuery for scans over millions of rows, object storage such as S3 for cheap files, Redis for 1-row lookups in under a millisecond. Storage is the role where the technology most often overlaps with a source. Postgres is a source for the shop and would be storage for a pipeline that fed a web application.
Consumer
A consumer is whatever reads the stored result and acts on it: a Looker dashboard, a machine learning model's training job, an export to an accounting system, another pipeline. The consumer is the reason the pipeline exists, and its question decides the shape storage holds. The shop's consumer asks for revenue by category by day, so that is the shape in Snowflake. A consumer asking for revenue by customer would need a different table, built from the same orders.
- orders (source, Postgres)
- clean_and_sum (transform, Python + SQL)
- daily_category_revenue (storage, Snowflake)
- Revenue dashboard (consumer, Looker)
- orders -> clean_and_sum
- clean_and_sum -> daily_category_revenue
- daily_category_revenue -> Revenue dashboard
| Role | Produces or holds | Shop example | How to recognise it |
|---|---|---|---|
| Source | Data as a by-product of business activity | Postgres orders table | Exists for an operational job; the pipeline only reads it |
| Transform | A new shape from an input shape | Python task that cleans and sums | Keeps nothing between runs |
| Storage | The result, ready to be read repeatedly | Snowflake daily_category_revenue | Chosen for the reads that will hit it |
| Consumer | A decision, a screen or an action | Looker chart read at 8:00 | The reason the pipeline exists |
A role is a position in the flow, and the same product plays different roles in different pipelines: Snowflake is storage here and a source for a pipeline that exports finance totals to an accounting system.
The shop adds a second pipeline that reads the Snowflake table daily_category_revenue every night and pushes category totals into NetSuite, the accounting system. In that second pipeline, which role does the Snowflake table play?
- Storage, because Snowflake is a warehouse
- Source, because the second pipeline reads from it and writes nothing back
- Consumer, because the table holds finished numbers
- Transform, because the totals were summed before they landed there
About this lesson
This section is part of the Pipeline Anatomy: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.