Data Lakehouse Architecture

Optimistic concurrency control. Writers create new data files and atomically update the metadata to point to them. Readers see a consistent snapshot. Failed writes leave no partial data.

Each write creates a new table version. Query any previous version by timestamp or version number. Enables reproducible analytics, debugging, and audit.

Add, rename, or reorder columns without rewriting existing data files. The metadata layer maps old files to the new schema using column IDs (Iceberg) or schema versioning (Delta).

Iceberg supports changing the partition scheme without rewriting data. Start with daily partitions; switch to hourly when volume increases. Old files keep their original partitioning; new files use the new scheme.

Every commit lands as a numbered JSON file (000...0.json, 000...1.json, ...) inside _delta_log next to the data. Each file lists Add and Remove actions plus optional Metadata, Protocol, and CommitInfo entries. Every ten commits Delta writes a Parquet checkpoint summarizing state so readers don't replay the full log. A reader lists _delta_log, finds the latest checkpoint, replays JSON entries after it, and knows which files belong to the current snapshot.

A single current metadata.json is the entry point. It records schema, partition spec, sort order, and pointers to each historical snapshot. Each snapshot points at a manifest list. Each manifest list enumerates manifests. Each manifest enumerates data files with column-level min/max stats. Three levels of indirection means the planner can prune entire manifest lists without opening a Parquet footer. The expensive part is keeping the tree healthy; the cheap part is querying it.

Hudi stores its transaction record as a sequence of instants under .hoodie/timeline. Each instant has a state (REQUESTED, INFLIGHT, COMPLETED) and an action (commit, deltacommit, compaction, clean, rollback). The instant carries metadata for the action so a reader can reconstruct table state at any point. This makes Hudi feel like a stream of changes rather than a series of snapshots, and is why incremental queries are first-class.

For Delta the table location plus _delta_log is enough. For Iceberg the catalog (Glue, Nessie, Polaris, Unity, Hive Metastore, REST) holds the pointer to the current metadata.json and is what makes atomic commits possible. Lose the catalog without backups and you've lost the table even if every Parquet file is still in S3.

A writer creates new data files in object storage. It then attempts to atomically update the metadata pointer (Delta log entry, Iceberg metadata file, Hudi timeline entry) to include the new files. If another writer committed first, the current writer retries with conflict resolution.

A reader reads the current metadata pointer, which gives it a consistent snapshot of the table. It sees only files committed before the snapshot. In-progress writes are invisible.

Delta uses S3's conditional PUT (or DynamoDB for coordination). Iceberg uses atomic rename of the metadata pointer through the catalog. Hudi uses a timeline with lock-free conflict detection. Different mechanisms, same result: writes are all-or-nothing.

Writers A and B both read snapshot 100, both stage files, both attempt to commit snapshot 101. The winner succeeds. The loser sees the snapshot advanced, replays its change against snapshot 101, tries again as 102. Non-overlapping partitions almost always succeed. Overlapping rows (two writers updating the same key): the loser fails with a concurrent modification exception. This is why most production lakehouses serialize writes to a single ingestion job per table.

Streaming ingest at one micro-batch per minute creates 1,440 tiny files per day per partition. Reads degrade because the planner opens hundreds of thousands of files and listing dominates wall clock. Fix: OPTIMIZE on Delta or rewrite_data_files on Iceberg, scheduled often enough that file count stays bounded.

A long-running batch job started at snapshot 100 still has file references when VACUUM 0 hours runs and removes everything not in the current snapshot. The reader fails with a file-not-found error halfway through. Fix: never set retention shorter than the longest possible read. Default Delta retention is seven days for exactly this reason.

Two MERGE INTO jobs target the same partition. Both read snapshot 100, both stage updates, the second fails with concurrent append exception. Fix: partition writers by key range, serialize MERGEs through a single orchestrated job, or accept the retry cost.

OPTIMIZE ZORDER BY (col1, col2, col3, col4, col5) on a 10 TB table runs for hours and produces marginal improvement. Z-order is most effective on the first one or two columns; later columns contribute almost nothing. Fix: pick the highest-cardinality predicate column you actually filter on and stop after one or two.

Every commit creates a new manifest list. After a million commits the metadata directory is tens of gigabytes and the planner slows even though the table is small. Fix: schedule rewrite_manifests and expire_snapshots on a cadence proportional to write rate.

A Merge-on-Read table accepts upserts as log files. If the async compaction service falls behind, reads must merge an ever-growing log against base files at query time and latency spikes. Fix: size the compaction service for peak write rate, not average, and alert on log-file count per partition.

Data stays in object storage. Transformations write back to the same storage layer. Eliminates an entire category of pipelines and their failure modes.

File compaction, orphan file cleanup, snapshot expiration, and data ordering (Z-order, sort order) are now the data engineer's responsibility. In a managed warehouse the vendor handles this. In a lakehouse you run OPTIMIZE and VACUUM (Delta) or rewrite manifests and expire snapshots (Iceberg).

A lakehouse catalog (AWS Glue, Nessie, Polaris, Unity Catalog) is the central registry mapping table names to metadata locations. Choosing and managing the catalog is a new infrastructure concern.

Since data is in an open format, different teams can use different engines: Spark for heavy transformations, Trino for interactive queries, Flink for streaming, Athena for ad-hoc. Powerful but requires understanding how each engine interacts with the table format.

Reality: it replaces the lake side of the lambda architecture. Warehouses still win on low-latency BI for many workloads, especially high-concurrency dashboards with sub-second budgets. Common shape: lakehouse for storage and transformation, warehouse as a fast serving layer fed from gold tables.

Reality: concurrency models differ, migration semantics differ, integrations differ, partition stories differ materially. Iceberg supports partition evolution natively; Delta does not without rewrite. Delta has deepest Spark/Databricks integration; Iceberg has broadest cross-engine story. Treating them as interchangeable misses the decision.

Reality: most table formats give snapshot isolation, not serializability. Two non-conflicting writers can commit against the same starting snapshot and both succeed. A reader sees a consistent snapshot, but two readers at different times can disagree about row counts without anyone violating ACID. Be precise.

Reality: mostly used for incident recovery (revert a bad batch), debugging (what did the table look like when this row was wrong), and ML training reproducibility (train against the snapshot the model was trained on). Production analytics queries almost always read latest. Building time travel into daily BI is a smell.

Reality: OPTIMIZE compacts small files into larger ones. That helps when your bottleneck is file count and listing overhead. Z-ORDER changes data layout so predicate pushdown can prune more aggressively — but only on the first one or two columns and only if you actually filter on them. Without the right keys, OPTIMIZE alone helps less than expected.