Data Lake or Warehouse?
Concepts covered: Data Lake Architecture
Three shapes, separated by who owns the files
A warehouse stores tables in its own internal format that only its engine reads, and in return gives transactions, row-level updates, indexes or clustering, and query performance that nobody has to tune at the file level. A data lake stores open files, Parquet or JSON, on object storage that any engine can read, charges the object store's rate for storage, and gives no transactions, no schema enforcement and no consistent view unless something adds them. A lakehouse is a lake with a table format from the previous section and a catalog on top, which restores the transactional properties on open files, so the engine choice stays open while the correctness returns.
| Axis | Warehouse | Data lake | Lakehouse |
|---|---|---|---|
| Who can read the files | the warehouse engine | any engine with a Parquet reader | any engine that speaks the table format |
| Transactions | full, row level | none without a table format | snapshot-level ACID from the log |
| Updates and deletes | in place, cheap | rewrite files by hand | rewrite files, tracked by the log |
| Raw and semi-structured data | loaded and typed first | stored as landed | stored as landed, promoted when typed |
| Storage price | the vendor's rate, often near object storage | object storage rate | object storage rate plus log and snapshot overhead |
| Compute price | billed by the vendor's unit on every read | whichever engine you run, including free ones | same as the lake |
| Operational burden | lowest | highest: compaction, statistics and consistency are yours | medium: the format handles consistency, compaction stays yours |
A scenario with 2 workloads
Place a company with 2 very different tables. The first is 500 TB of raw clickstream, retained 3 years, read mostly by model training jobs that scan months of Parquet, with an analyst querying it a few times a week. The second is 2 TB of curated finance marts, read by 400 dashboard users who expect sub-second filters and by an audit process that needs the numbers as they stood on the last day of each quarter. Loading all 500 TB into the warehouse means paying its storage rate on 500 TB and its compute on every training scan, with the training framework reading through a connector instead of opening files. Keeping the finance marts as bare Parquet in the lake means 400 users hitting files with no indexes and an audit that cannot reconstruct a quarter end.
The design that fits puts the clickstream in the lake as raw JSON, promotes it to Parquet under a table format for the curated layer, and serves the finance marts from the warehouse or from a gold lakehouse layer with time travel enabled, depending on which engine the dashboard tool already uses. The layers also answer the retry and late-data probes: the curated layer overwrites a partition per run, so a retried run produces the same partition, and a late event lands in the partition of its event date and triggers a rewrite of that day.
- App events (source, JSON, as emitted)
- Billing DB (source, nightly extract)
- Raw landing (bronze, object storage, as landed); freshness: 15 min; retries: 3; idempotency: file name = batch id
- Curated events (silver, Parquet + table format); freshness: hourly; backfill: rewrite by event_date; idempotency: overwrite partition
- Finance marts (gold, warehouse or gold table); freshness: daily 06:00; alerting: row count vs prior day
- Model training (consumer, scans months of Parquet)
- Dashboards (consumer, 400 users)
- App events -> Raw landing
- Billing DB -> Raw landing
- Raw landing -> Curated events
- Curated events -> Finance marts
- Curated events -> Model training
- Finance marts -> Dashboards
- Sources: App events, Billing DB
- Raw: Raw landing
- Curated: Curated events
- Serving: Finance marts
- Consumers: Model training, Dashboards
Where the lakehouse pays off
The lakehouse earns its place when the same 500 TB has to serve both audiences: the training jobs keep opening Parquet directly, and an analyst gets a consistent snapshot, a schema enforced on write and a quarter-end version to query, without a second copy in the warehouse. The price is operational: compaction schedules, statistics collection and retention windows are the team's job, where a warehouse would run them unseen. A design that chooses lakehouse should name who owns those 3 jobs.
The questions that decide it
- Who reads it, how often, and with what latency expectation: 400 users at sub-second or 3 jobs at hours
- How big it is in TB, because storage rate differences multiply by that figure
- Whether rows mutate after landing or only append, since updates on bare files mean rewrites
- How many engines need the same table, since a proprietary format stops at 1
- Whether an auditor needs the table as it stood on a date, which requires a log or a warehouse
- Who will run compaction and statistics if the files are yours
About this lesson
This section is part of the The Storage Question: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.