Delta or Iceberg?
Concepts covered: Table Formats
The problem a directory cannot solve
A partitioned directory of Parquet files has no definition of the table other than whatever a list call returns at the moment it runs. That produces 4 failures the interviewer will probe. A writer that dies after uploading 3 of 8 files leaves 3 orphans that every reader counts. Two writers that overwrite the same day each believe they finished, and the last upload wins silently. A reader that starts during a rewrite sees old files for one partition and new files for another. And the schema is whichever footer the engine opens first, so a column added last week reads as null or as an error depending on file order.
Both Delta Lake and Iceberg answer with the same idea: a log, stored next to the data, that records which exact files make up each version of the table. Readers consult the log and never list the directory. Writers add files, then commit by writing one new log entry, and the commit either lands whole or fails. Everything the interviewer calls ACID, time travel and schema evolution follows from where that log lives and what it records.
How each format writes the log
Delta Lake keeps a _delta_log/ directory of numbered JSON commits, 00000000000000000041.json for commit 41, each listing the files added and removed and any schema change. Every 10 commits it writes a Parquet checkpoint of the full state so readers replay only the tail. A commit is the atomic creation of the next number; if 2 writers race for 41, the object store accepts 1 and the other rereads the log and retries as 42.
[
{"commitInfo": {"timestamp": 1792483200000, "operation": "WRITE", "operationParameters": {"mode": "Overwrite", "partitionBy": "[\"event_date\"]"}}},
{"remove": {"path": "event_date=2026-10-20/part-00000-a1.parquet", "deletionTimestamp": 1792483200000, "dataChange": true}},
{"add": {"path": "event_date=2026-10-20/part-00000-b7.parquet", "partitionValues": {"event_date": "2026-10-20"}, "size": 134217728, "dataChange": true, "stats": "{\"numRecords\":1200000,\"minValues\":{\"user_id\":1041},\"maxValues\":{\"user_id\":49998871}}"}}
]Iceberg keeps a tree. A metadata.json file names the current snapshot and the schema and partition spec history; each snapshot points to a manifest list; each manifest lists data files with per-file column statistics. A commit swaps the pointer to the new metadata file in the catalog, 1 atomic operation, and a loser in a race rebases on the winner's snapshot and retries. Because statistics live in the manifests, planning a query reads a few metadata files rather than 73,000 directory listings.
{
"snapshot-id": 8412030719,
"parent-snapshot-id": 8412030718,
"timestamp-ms": 1792483200000,
"manifest-list": "metadata/snap-8412030719.avro",
"summary": {
"operation": "overwrite",
"added-data-files": "8",
"deleted-data-files": "8",
"added-records": "9600000",
"deleted-records": "9500000"
}
}Three axes, compared
| Property | Delta Lake | Iceberg |
|---|---|---|
| Atomic commit | create the next numbered log file; collisions retry | swap the catalog pointer to a new metadata file; collisions rebase and retry |
| Isolation | readers pin to the log version they started on | readers pin to the snapshot they started on |
| Time travel | by version number or timestamp, back to the oldest retained log entry | by snapshot id or timestamp, back to the oldest retained snapshot |
| Retention | VACUUM removes files older than the retention window, 7 days by default | expire_snapshots removes snapshots and the files only they reference |
| Schema evolution | add, rename and drop; rename and drop require column mapping mode so files are matched by id rather than name | columns carry ids from creation, so add, rename, drop and reorder never touch data files |
| Partition evolution | partition columns are physical; changing them rewrites the table | partition spec is versioned; new data uses the new spec, old files keep theirs, queries plan across both |
| File statistics | per file in the log, first 32 columns by default | per file in manifests, configurable per column |
Making the choice in the room
The interviewer wants a decision tied to properties, and the properties come from the workload. Three questions settle it. How many engines read and write the table? Pick the format every engine in the stack already supports natively, because a format one engine writes and another cannot read is a copy job waiting to happen. Will the partition scheme change? A table that starts partitioned by day and needs hours 6 months later rewrites 9 TB under Delta Lake and changes a spec under Iceberg. How often does the schema change by rename? Iceberg's column ids make it safe by default; Delta Lake needs column mapping enabled before the first rename.
Then name the cost both formats share, because the interviewer will ask. Every rewrite keeps the old files until retention expires them. The events table rewrites 3 days of late-arriving partitions each morning, 150 GB of new files a day, and with a 30-day retention window the old versions hold 4.5 TB that exist for time travel alone. That number reappears on the bill in the last section.
About this lesson
This section is part of the The Storage Question: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.