Why Pipelines Exist
Concepts covered: ELT vs ETL
Every company that runs software produces data in one shape and needs it in a different shape, in a different place, on a different schedule. That gap is the entire reason data engineering exists. The gap is not a bug. It is structural. Operational systems are built to handle one user at a time, fast, with strict consistency. Analytical systems are built to scan billions of rows, slow per row, with relaxed consistency. The two are different machines optimized for different jobs.
Three Gaps That Force a Pipeline
| Gap | What It Means | Concrete Example |
|---|---|---|
| Location | Data is created in one system, needed in another | App writes to Postgres; analyst reads from Snowflake |
| Shape | The shape that is fast for the writer is slow for the reader | Normalized rows for transactions; wide denormalized columns for dashboards |
| Time | Data is produced continuously; reports want a daily or hourly snapshot | Click events stream all day; the marketing team wants a 9am summary |
A pipeline closes all three gaps at once. It moves data from where it lives to where it is needed. It reshapes the data along the way. It coordinates the timing so the consumer sees a consistent picture rather than a constantly shifting one. Anything that closes those gaps is, in some sense, a pipeline. A nightly bash script that copies a CSV file from one server to another is a tiny pipeline. A platform like Airbnb's that runs tens of thousands of orchestrated jobs is a very large pipeline. The mechanics scale; the underlying job does not change.
Without a Pipeline
Consider a startup with no pipeline. The CEO asks for revenue by region. An engineer SSHes into the production database and runs a SELECT. The query takes the production database to its knees, the app slows to a crawl, and customers notice. The next time the question comes up, the engineer extracts the data manually to a spreadsheet, where it becomes stale within hours. The third time, someone writes a script that runs at 3am. The script is a pipeline, even if nobody calls it that yet.
- Without a Pipeline
- Analytics queries hit production and slow the app
- Every report is a one-off manual extract
- Data is stale by the time it is read
- No one knows which numbers are authoritative
- With a Pipeline
- Production stays fast; analytical work runs on a copy
- Reports run on demand from a prepared dataset
- Freshness is explicit and known (last hour, last day)
- One pipeline produces the canonical numbers; debates end
The Smallest Possible Pipeline
# A real pipeline in three lines of bash:
pg_dump --table=orders production_db > orders.sql
psql analytics_db < orders.sql
echo "Extract complete at $(date)" >> /var/log/pipeline.logThis script does the three things a pipeline does. It reads from a source (production_db). It moves data toward a destination (analytics_db). It records that the run happened. It is missing scheduling, error handling, transformation, and any concept of incremental updates, but it is a pipeline. Everything in the rest of this lesson, and everything in the rest of this curriculum, is a refinement of what those three lines are trying to do.
If a question can be answered by a single SQL query against the existing database, a pipeline is overkill. Pipelines exist because most real questions cannot.
- A pipeline closes the gap between where data is produced and where it is needed.
- Three gaps drive the need: location, shape, and time.
- Every script that copies and reshapes data on a schedule is a pipeline, no matter how small.
About this lesson
This section is part of the Pipeline Anatomy: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.