The Data Lake
A data lake is files in object storage. Object storage is a service that keeps each file as an object under a key that looks like a path. It writes an object whole and can read back all of it or a part, but it never edits an object in place: to change a file you write a new one. Its price per terabyte is a small fraction of a database disk, and it holds any format you hand it, so the lake becomes the place a company keeps everything it might want later.
| Zone | Example path | Format | Written by |
|---|---|---|---|
| Raw | lake/orders/raw/2026-10-22/events-0001.json | JSON, exactly as received | Ingestion pipeline |
| Cleaned | lake/orders/cleaned/2026-10-22/part-0001.parquet | Parquet, typed, 1 record per event | Cleaning job |
| Curated | lake/orders/curated/daily_city_revenue/2026-10-22/part-0000.parquet | Parquet, aggregated for a use | Modelling job |
Zones
A lake without structure turns into a swamp nobody trusts, so teams divide it into zones with 1 rule each. Raw holds data exactly as it arrived and is never changed, so the original is always there to go back to. Cleaned holds the same records with types applied, broken records set aside and duplicates removed. Curated holds data shaped for 1 consumer, such as daily revenue per city or a feature file for the delivery time model. Names vary between companies, but the 3 jobs stay the same.
- Order events (source, app emits JSON)
- Raw zone (bronze, JSON files, never edited)
- Cleaned zone (silver, Parquet, typed)
- Curated zone (gold, Parquet, aggregated)
- Warehouse load (consumer, nightly batch)
- Model training (consumer, weekly)
- Order events -> Raw zone
- Raw zone -> Cleaned zone
- Cleaned zone -> Curated zone
- Curated zone -> Warehouse load
- Cleaned zone -> Model training
- Sources: Order events
- Lake: Raw zone, Cleaned zone, Curated zone
- Readers: Warehouse load, Model training
The raw zone justifies its storage bill the first time something goes wrong upstream. In June the app shipped a version that wrote total_cents in dollars for 3 days, so every cleaned file for those days held totals 100 times too small. The raw events were untouched. The team fixed the cleaning rule and reran it over those 3 days, which rebuilt the cleaned and curated zones with the right totals. Had the lake only kept cleaned data, the original amounts would have been gone.
Why Parquet
Raw files arrive in whatever format the source speaks: CSV from a partner, JSON from the app. From the cleaned zone on, the lake converts to Parquet, which applies the warehouse's columnar idea inside a file.
| Format | Layout inside the file | Types | Reading 3 of 40 fields |
|---|---|---|---|
| CSV | 1 line per record, fields separated by commas | None, everything is text | Reads every byte of every line |
| JSON | 1 object per record, field names repeated in each | None enforced | Reads and parses every byte of every object |
| Parquet | Values grouped by column, with a footer describing the file | Written into the file | Reads the 3 column groups and skips 37 |
Take 1 day of 2 million order events with 40 fields each. As JSON every record repeats all 40 field names, so at roughly 600 bytes per record the day comes to around 1.2 GB. Parquet writes each field name once per file and compresses similar values stored next to each other, so the same day commonly lands at a quarter of that size or less. A query that reads 3 fields then touches 3 small column groups instead of 1.2 GB of text, and takes each field's type from the footer.
- Do
- Land data in raw exactly as received, in its original format
- Convert to Parquet in the cleaned zone and read Parquet from there on
- Name zones so a path tells you what has been done to the data
- Avoid
- Edit a file in the raw zone, ever
- Run daily reports against JSON or CSV when a Parquet copy exists
- Skip zones and write a consumer's table straight from raw
A data scientist wants 18 months of order events to test a delivery time model, including 3 fields the cleaning job drops because no dashboard uses them. Where should she read?
- The raw zone, because it keeps every field of every event exactly as it arrived.
- The cleaned zone, because it holds every event as typed Parquet for fast scans.
- The warehouse orders table, because it holds 18 months of history with enforced types.
- The operational database, because it is the original source of every order.
About this lesson
This section is part of the Storage Layers: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.