Iceberg vs Delta Lake vs Hudi

Both formats use optimistic concurrency. A writer reads the table, writes its new data files without taking a lock, and only at commit time checks whether anyone else got there first. Writers never block each other while they work, and a conflict costs a retry or a failed job rather than a corrupted table.

Take 2 writers that both read the table at version 7. Each prepares an append. Writer A commits first. In Delta, A creates 00000000000000000008.json. In Iceberg, A swaps the catalog's pointer from the version 7 metadata file to its new one. Writer B now tries the same step from a stale base. In Delta, creating version 8 fails because it exists; in Iceberg, the swap fails because the pointer no longer holds the value B expected.

B doesn't simply fail. It checks what A changed. The Iceberg spec says the writer "must retry the update based on the new current version", and some operations can be re-applied when well-defined conditions hold. 2 appends commute, so B re-applies on top of A and lands at version 9. Delta does the same check against the files and partitions B read: an operation that read a file A deleted fails with ConcurrentDeleteReadException, one that read a partition A appended to can fail with ConcurrentAppendException, and 2 plain inserts cannot conflict.

Where the 2 diverge is what makes the commit atomic. Iceberg needs an atomic compare-and-swap, which the catalog provides, so the catalog is part of the correctness story and not only a name lookup. A Delta commit on plain object storage needs the store to create a log file only if it is absent, and catalog-managed Delta tables move that ratification into a catalog as well. If an interviewer asks what makes a lakehouse commit atomic, the answer is that data files are immutable and one small metadata step either happens or does not.

Isolation is the other axis. Delta documents 2 levels, with WriteSerializable as the default on Databricks and Serializable available per table, and with row-level conflict detection when deletion vectors are on, so 2 writers touching different rows of one file need not conflict. Iceberg's spec bases serializable isolation on the metadata swap, and its engines decide how finely they detect conflicts. Partitioning your writes so concurrent jobs touch different partitions reduces conflicts in both.

Iceberg treats partitioning as table configuration, not as extra columns. Declaring day(ts) or bucket(16, id) on a source column is enough, and Iceberg derives the partition value itself. Because readers filter on the real column, a query on ts prunes partitions without anyone writing a ts_day predicate. The spec lists 8 transforms: identity, bucket, truncate, void, and the time ones year, month, day and hour.

The larger benefit is evolution. Changing a table's partition spec is a metadata operation that does not rewrite files, so old data keeps the old layout and new data uses the new one. A table partitioned by month while it was small can move to day as it grows, and a query plans each layout with its own derived filter.

Delta Lake partitions by explicit columns and offers liquid clustering as the alternative. You declare CLUSTER BY with up to 4 columns, and OPTIMIZE clusters data incrementally. Changing the keys does not rewrite existing data, which gives Delta the same cheap-to-change property by a different route. A clustered table can't also use partitioning or ZORDER, so the choice is per table.

The decision rule: low-cardinality, time-shaped access favours Iceberg's transforms, which are portable across engines, while multi-column filter patterns that change over time suit Delta's clustering. In either, avoid partitions so fine that each holds a few small files, because small files cost more in metadata and listing than pruning saves.

Parquet files are immutable, so deleting 1 row means one of 2 things: write a new file without it, or write a note saying the row is gone. The first is copy-on-write, the second merge-on-read.

The cost gap is easy to see. Say a 128 MB file holds 1,000,000 rows and a job deletes 1 of them. Copy-on-write rewrites the file, which writes 128 MB and 999,999 rows to remove 1. A merge-on-read delete records 1 deleted position, and the file stays as it is. The price moves to the reader, which must apply that record, and to maintenance, which eventually folds the deletes in.

Iceberg v2 introduced delete files for this. A position delete file names a data file and the row position in it; an equality delete file identifies rows by column values, which suits streaming upserts where the writer does not know the file. Sequence numbers decide which delete files apply to which data file. Delta Lake commits remove and add whole files, or mark rows through deletion vectors.

Both formats let you choose per table. Copy-on-write keeps reads fast and suits tables that are read far more than they're changed. Merge-on-read keeps writes fast and suits frequent small changes such as CDC, at the cost of read-time merging until compaction runs.

A deletion vector stores the deleted row positions of one data file as a compressed Roaring bitmap. Delta Lake has them behind the deletionVectors table feature, which needs reader version 3 and writer version 7. The property delta.enableDeletionVectors turns them on, and enabling it upgrades the table protocol, so older Delta clients can no longer read the table.

Iceberg added the same idea in format version 3. A deletion vector is a deletion-vector-v1 blob in a Puffin file, built on 32-bit Roaring bitmaps, and a snapshot may hold at most 1 per data file. The Iceberg spec also says position delete files must not be added to v3 tables, though existing ones stay valid, so on v3 the vector replaces the position delete file for new work.

In both, soft deleted rows stay in the data file until a rewrite. In Delta, OPTIMIZE rewrites the files, as does REORG TABLE ... APPLY (PURGE), and old files linger until VACUUM. In Iceberg, compaction rewrites data files and expiring snapshots lets old files go. The vector shrinks the cost of the delete itself, not the cost of the cleanup.

Check support before you commit to the feature. A v3 table needs engines and catalogs that implement v3, and a Delta table with deletion vectors is not translated by every interoperability tool.

Not natively, but the data files are the same Parquet either way, so what must be translated is the metadata. 2 tools do that. Delta UniForm generates Iceberg metadata after each Delta commit, so one copy of the Parquet files serves both. Iceberg and Hudi clients then read the table and can't write to it, and the documentation warns that an outside writer can destroy the Delta table.

Apache XTable is the format-neutral route. It is not a new format. It reads a table's metadata and writes metadata for another format into the same base path, among those 3 formats. It's an incubating Apache project, and its documented limits include Delta deletion vectors and merge-on-read tables, so check which features your table uses against its supported list.

The pattern most teams settle on is to write in one format and expose the table to other engines through a catalog, not to dual-write. Pick the writer format by who writes the table, and treat the other formats as read paths. Also look at the catalog: it decides discovery and access control, and for Iceberg and catalog-managed Delta it also decides commit atomicity.

Hudi brings database-style upserts and deletes to data lakes and keeps a timeline of every action on a table under .hoodie. Each action moves from requested to inflight to completed, and the current state is the highest state recorded. That timeline plays the role of Iceberg's metadata tree and Delta's log.

Its distinguishing feature is indexing. To upsert a record, the writer must find which file holds that record. Hudi's indexes exist to make that lookup cheap: a record index that maps a record key to its location, plus bloom filter and bucket indexes, and secondary and expression indexes. Without one, finding the files to rewrite means scanning the dataset.

Hudi's 2 table types map to the delete trade-off above. Copy-on-write rewrites base files on every update. Merge-on-read appends changes to row-based log files and a periodic compaction folds them into new base files, which lowers write latency and raises query latency until compaction runs.

In Hudi vs Iceberg vs Delta comparisons, Hudi wins when the workload is high-frequency upserts or CDC ingestion into a lake and you want the index as a first-class part of the table. It loses on engine support: Iceberg and Delta are the formats most engines and managed services target. Delta UniForm lists Hudi output as a preview, so treat Hudi readers of a Delta table as experimental.

A table format moves cost from write time to maintenance, and every format needs a schedule for it. Small files come from frequent commits and slow every read, so each has a compaction path. Delta uses OPTIMIZE and Iceberg has a compact data files action, while Hudi runs compaction and clustering services.

Old snapshots are the second job. Time travel keeps old data files and metadata alive, so Iceberg's recommended maintenance is to expire snapshots, remove old metadata files and delete orphan files, and Delta's is VACUUM. Delete too eagerly and you lose time travel and break long-running readers. Delete too late and storage grows with every rewrite.

Plan this before launch. A common production failure is a table whose metadata grew without bound because nobody scheduled the cleanup, and it is a fair system design question about any lakehouse pipeline.

Start with where the table is written and read, because engine support outweighs any single feature. If your jobs run on Databricks or Spark and the table mostly lives there, Delta Lake is the lowest-friction choice. If the table must be shared by Flink and Trino, or by Snowflake alongside other engines, Iceberg's engine-neutral spec and catalog model fit better.

Then look at the write pattern. Append-heavy tables with occasional corrections suit either format. A table that receives a steady stream of row updates from CDC deserves a closer look at Hudi, or at merge-on-read with deletion vectors on Iceberg v3 or Delta, and a test of read latency before compaction.

Last, price the exit. Because all 3 store Parquet, switching is a metadata problem and tools exist for it, so no choice is permanent. That makes the right move to choose for the next 2 years of your platform and avoid anything built around a single feature that has no equivalent elsewhere.

Interviewers rarely ask you to recite feature lists. They ask what problem a table format solves, why it makes object storage transactional, and how you would handle a late-arriving correction or a GDPR delete on a petabyte table. The answers above carry most of it: immutable files, a metadata step that is atomic, and a choice between rewriting files and recording deletes.

Expect follow-ups on the failure modes: what happens when 2 jobs write at once, why small files hurt, and what you would schedule to keep the table healthy. The data engineer interview prep path covers those topics. For Spark reads and writes on these tables, use the PySpark interview questions and the PySpark practice problems.

If your target team runs on Databricks, Databricks interview questions cover OPTIMIZE and VACUUM on Delta, and clustering as well.