A curated lake table is plain Parquet in object storage with two concurrent Spark writers committing

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A curated lake table is plain Parquet in object storage with two concurrent Spark writers committing into overlapping partitions. Readers see partial writes, last-writer-wins corrupts the table, and there is no rollback. Apply the section's lakehouse framing and replace the plain Parquet table with an open table format that adds the metadata layer; the data files stay Parquet, only the metadata changes.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.