The Four Roles in Any Pipeline

Concepts covered: ELT vs ETL

Every pipeline, no matter how complex, can be described in terms of four roles. A source produces data. A transform reshapes it. Storage holds it for later. A consumer reads it for some purpose. Real pipelines often have many of each, chained together, but the roles do not change. Naming the four roles is the single most useful skill a new data engineer can develop, because once they are named, every architecture diagram becomes legible.

Role 1: Source

A source is wherever data originates. It is the system that produced the data in the first place. The source is upstream of everything else. A pipeline does not own its sources; it consumes from them. This distinction matters because the source can change without warning, and the pipeline must absorb that change. Common sources include operational databases (Postgres, MySQL), event streams (Kafka, Kinesis), third-party APIs (Stripe, Salesforce), application logs (CloudWatch, Datadog), and file drops (an FTP server, an S3 bucket where a partner deposits CSVs).

Role 2: Transform

A transform takes data in one shape and produces data in another. The work spans a wide range. Cleaning a phone number into a standard format is a transform. Joining two tables to produce a denormalized fact table is a transform. Aggregating a billion events into a thousand daily summaries is a transform. The defining feature of a transform is that the output shape differs from the input shape. Transforms can be written in SQL, Python, Spark, dbt, or anything else that can read data and write data. The language is a tool choice; the role does not change.

Role 3: Storage

Storage is the layer that holds data between steps. It is the resting place. Storage is what makes pipelines durable: if the next step fails, the data does not have to be re-fetched from the source, because it is sitting safely in storage. Common storage layers include data warehouses (Snowflake, BigQuery, Redshift), data lakes (S3, GCS, ADLS), and operational databases when used as a destination rather than a source. Storage and source can be the same physical system in different roles. A Postgres database is a source for the pipeline that pulls from it and a storage layer for the pipeline that writes to it.

Role 4: Consumer

A consumer is anything downstream that reads the prepared data and uses it. Consumers include dashboards (Looker, Tableau, Mode), machine learning training jobs, reverse-ETL tools (which push curated data back into operational systems like Salesforce or HubSpot), internal applications that show data to users, and humans running ad-hoc SQL queries. The consumer is the reason the pipeline exists. A pipeline with no consumer is, by definition, dead code. Designing the consumer-facing shape first and working backward is the more common pattern; designing the source-facing shape first and hoping it works for consumers is the more common mistake.

• Source: Owned by another team or vendor. Cannot be controlled, only consumed. Postgres, Kafka, Stripe API, S3 file drops.
• Transform: The work the pipeline does. SQL, Python, Spark, dbt. Cleaning, joining, aggregating, deduplicating.
• Storage: Durable layer between steps. Snowflake, BigQuery, S3. Survives transform failures so retries do not re-fetch.
• Consumer: Dashboards, ML jobs, reverse-ETL, applications. The shape consumers need drives the design.

All Four Roles in One Sentence

A pipeline reads from one or more sources, applies one or more transforms, lands the result in storage, and serves consumers. That sentence describes a script someone wrote in 2008 and a modern lakehouse running on Databricks. The roles are invariant; only the tools change.

[Stripe API]  --(source)-->  [Python script]  --(transform)-->  [Snowflake table]  --(storage)-->  [Looker dashboard]  --(consumer)
   source              transform                  storage                                consumer

Do

• Name the four roles for any pipeline before adding detail
• Treat sources as untrusted: their schema and timing can change
• Pick the storage layer based on how the consumer will read it

Avoid

• Confuse storage with source (the same Postgres table can be either, depending on the pipeline's role)
• Skip storage between transforms in long pipelines (failures lose work)
• Build pipelines without a named consumer in mind

About this lesson

This section is part of the Pipeline Anatomy: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.