The Lakehouse: ACID on Object
Object storage offers 4 operations that matter here: put an object, get it, list keys under a prefix, delete it. Each put is atomic, but nothing groups 2 puts into 1 change, and most object stores implement a prefix 'rename' as a copy plus a delete for every key, so moving 500 files reaches readers as 1,000 separate changes.
A plain data lake defines a table as every file under a directory. That definition fails in 4 ways once some jobs write while others read:
- Torn reads: a query that lists the directory mid-write counts some new files beside some old ones.
- Orphaned output: a job that dies after writing 30 of its 50 files leaves 30 files that every later query counts.
- Lost history: an overwrite deletes the old files, so nothing records what the table held yesterday at 09:00.
- Slow planning: listings commonly return at most 1,000 keys per request, so a table of 1 million files costs 1,000 round trips before any data is read.
The metadata layer
A lakehouse keeps the same columnar data files and adds a second kind of file: metadata naming exactly which data files form the table. A root metadata file holds the schema and the partition spec, and it also lists the table's snapshots. Each snapshot points to manifests, and each manifest lists data files. For every file the manifest records its row count and partition values, along with a min and max per column. A catalog stores 1 thing per table: the location of the current root metadata file.
- Writer job (transform, batch or streaming job)
- Data files (storage, immutable columnar files)
- Metadata files (storage, snapshots, schema, file stats)
- Catalog pointer (custom, 1 compare-and-swap per commit)
- Query engine (consumer, reads the pointer first)
- Writer job -> Data files
- Data files -> Metadata files
- Metadata files -> Catalog pointer
- Catalog pointer -> Query engine
One commit, traced
The orders table is partitioned by order_date and sits at snapshot 41, made of 3 files, f1 to f3, where f3 holds 2026-10-23. A nightly job recomputes that date and must replace f3 with 2 corrected files. The catalog accepts a new pointer only if it still holds the value the writer last read: a compare-and-swap.
| Step | What the job writes | Snapshot a reader gets |
|---|---|---|
| 1 | data files f4, f5 | 41: f1, f2, f3 |
| 2 | metadata m42 listing f1, f2, f4, f5 | 41: f1, f2, f3 |
| 3 | pointer swapped from m41 to m42 | 42: f1, f2, f4, f5 |
| Crash before 3 | f4, f5 and m42, referenced by nothing | 41, unchanged |
Atomicity comes from the single swap: every reader sees snapshot 41 or snapshot 42, with no state between them. Isolation comes from immutability, since no commit edits a file another query might be reading. Durability belongs to the object store. The job never deleted f3: snapshot 41 still lists it, and a cleanup step removes it only after 41 expires.
A job writes 40 of its 50 new data files into the orders location, then its machine dies before step 2. A dashboard queries orders a minute later. What does the dashboard read?
- The 40 new files plus the old ones, because the files already sit inside the table's location
- Nothing until cleanup runs, because the crashed job left the table locked against readers
- Exactly the previous snapshot, while the 40 files sit unreferenced until a cleanup removes them
- The previous snapshot, but only after the engine finds and deletes the 40 partial files
Why the metadata is the product
- Directory of files
- The table is whatever a listing returns right now
- A half-finished write is visible to readers
- Schema is inferred from whichever files get opened
- Pruning needs a listing, then every file's footer
- Lakehouse table
- The table is the file list in the current snapshot
- A commit appears all at once or not at all
- Schema is stored once, with a permanent id per column
- Pruning reads min and max values from manifests
Copy every data file of a lakehouse table into a fresh bucket and leave the metadata behind: what you hold is a pile of files that has lost its schema and its version history, and nothing says which files were ever committed. Every capability in the rest of this lesson is a change to metadata. The data files are open columnar files any engine can read; the metadata turns them into a table many jobs can share.
The orders table holds 2,000 data files across 3 years. A query filters order_date = DATE '2026-10-23'. How does the engine choose which files to open?
- It reads partition values and min/max stats per file from the manifests, then opens only matching files
- It lists every prefix under the table location, then opens each file footer to compare min/max values
- It opens all 2,000 files and relies on predicate pushdown to discard row groups for other dates
- It asks the catalog, which keeps a per-date index of the files committed to each table
About this lesson
This section is part of the Storage Layers: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.