Storage Is Not Just the Database
In Pipeline Anatomy the storage role was a single box: the place data lands so a consumer can read it. Open that box at any company past its first year and you find several systems holding the same data in several shapes. A food delivery app taking 2 million orders a day keeps each order in 3 places, because 3 different readers need it in 3 different forms.
The checkout screen needs every field of 1 order right now, and it has to change that order the moment a courier picks it up. An analyst needs 60 million orders from last month, 3 columns each, summed by city. A data scientist needs every raw order event from the last 2 years, including the fields nobody has cleaned yet, to test a delivery time model. No single layout on disk serves all 3 reads well, so storage splits into shapes.
| Shape | What it holds | Who reads it | Typical read |
|---|---|---|---|
| Operational database | The live orders, 1 row per order, changed as the order moves | The app itself | Fetch or update 1 order by its id in a few milliseconds |
| Data warehouse | Cleaned, typed tables of orders and customers | Analysts, dashboards | Sum 60 million orders by city in seconds |
| Data lake | Raw files of every order event as received, plus cleaned copies | Pipelines, data scientists | Reread 2 years of files with a new rule |
One order in 3 shapes
Here is order 8841207 as the operational database holds it: 1 row, every column, found by its id.
| order_id | customer_id | city | total_cents | status | placed_at |
|---|---|---|---|---|---|
| 8841207 | 55018 | Austin | 2499 | delivered | 2026-10-22 18:41:03 |
The lake holds the same order as the events the app emitted while the order moved: 1 JSON object per event, written into a file alongside thousands of neighbours and never edited again. The first event looks like this.
{
"event": "order_placed",
"order_id": 8841207,
"customer_id": 55018,
"city": "Austin",
"total_cents": 2499,
"ts": "2026-10-22T18:41:03Z",
"app_version": "7.12.0"
}The warehouse holds it as a slice through a column. Every order's total_cents sits together on disk, every city sits together, and a query summing revenue by city reads those 2 slices and touches nothing else. Order 8841207 is in there, but a query asking for it by id has to search a column of 1.4 billion ids to find it, then gather its other fields from 5 more columns. The layout that makes sums fast makes a single lookup slow, and that trade is the whole reason the shapes differ.
Why one shape cannot do every job
A disk hands back bytes in order, so whatever sits next to each other on disk is cheap to read together. Put a whole row together and 1 lookup is cheap, while a sum over 1 column drags every other column along for the ride. Put a whole column together and the sum is cheap, while 1 lookup has to visit every column separately. Files in a lake sit in a third arrangement: whole files written once. They are cheap to store and to reread end to end, but there's no index to find 1 record. Each arrangement wins 1 kind of read by giving up the others.
Keeping 3 copies of every order sounds wasteful until you price the alternatives. Storage for 1.4 billion orders costs less per month than 1 engineer's afternoon, while a checkout that waits behind an analyst's query costs orders. A pipeline's job is often to carry data from one layer to the next: out of the operational database, into the lake as files, then into the warehouse as tables. The batch versus streaming choice sets how often that movement runs; this lesson sets where the data lands.
The marketing team wants average order value per city for the last 90 days, refreshed every morning. Which storage shape should the query run against?
- The operational database, because it holds the most current orders with every column present.
- The data warehouse, because the daily query averages total_cents by city, reading 3 columns across millions of rows.
- The data lake's raw zone, because the JSON files hold every order event ever emitted.
- A summary table inside the operational database, loaded each night with the 90 day totals.
About this lesson
This section is part of the Storage Layers: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.