Storage Architecture

Concepts covered: Medallion Architecture, Columnar vs Row Formats, Table Formats, Partitioning Strategies, Small File Problem

"I would put it in S3" names a bucket rather than a layout. The interviewer needs 4 decisions, each with a cost at 40 million events a day: the layers, the file format in each, the table format that adds transactions, and the partition key.

3 layers and an immutable bottom

Bronze holds data as it arrived, duplicates included, append-only and never edited. When a transform bug miscounts refunds in gold, the fix is a rerun of silver and gold over bronze, with no second extraction from Stripe and no replay of WAL the slot released weeks ago. That rebuild is why the layers exist, and naming it tells the interviewer you have repaired a pipeline rather than only drawn one.

Parquet where it is read, Avro where it is written

Parquet stores each column contiguously with min and max statistics per row group, so a query reading 3 of 40 event columns skips the other 37 and opens only row groups whose range covers the filter. A day of 40 million JSON events at about 1 KB each is 40 GB raw and near 4 GB as Snappy Parquet. Avro is row-oriented and carries its schema, which suits the Kafka path: Debezium publishes Avro against the Confluent Schema Registry, so a new column on orders becomes a new schema version. Zstd compresses about 15% tighter than Snappy for a little more decode CPU, and silver takes that trade because Athena bills bytes scanned, so every read gets 15% cheaper; in bronze, read only for rebuilds, the gain is storage alone.

A table format on top of the files

Raw Parquet in S3 is a folder of files with no transactions: a Spark job that dies after 300 of 500 files leaves 300 files that look complete to every reader. Iceberg adds a metadata layer, as do Delta Lake and Hudi: a write produces new files and a new snapshot, and the table's current pointer moves to it in 1 atomic swap, so readers see the whole commit or none. Snapshot history also gives time travel: FOR VERSION AS OF reads gold as it was before yesterday's bad run, the first thing finance asks for during an incident.

The follow-up is always "what if the write fails halfway?" The snapshot was never committed, the pointer never moved, and a maintenance job removes the orphan files later. Saying "pointer" shows the mechanism behind the feature list, which "Iceberg gives you ACID" does not.

Partitioning by what the query filters on

The operations query filters by time and city; finance filters by day. Partition silver events by day of event_ts, so a day's query opens 1 of 365 partitions. Over a year the table is about 1.5 TB of Parquet, and at Athena's $5 per TB scanned a 1-day query costs roughly 2 cents while the same query on an unpartitioned copy costs over $7. At 200 analyst queries a day that is $1,400 against $4, and the interviewer wants that subtraction done at the whiteboard.

CREATE TABLE silver.events (
    event_id    STRING,
    user_id     BIGINT,
    city        STRING,
    event_type  STRING,
    event_ts    TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(event_ts));

ALTER TABLE silver.events
WRITE ORDERED BY city, event_ts;

"Partition by the filter column" leads candidates to over-partition. By user_id, each 5-minute commit writes a file for every active user among 3 million: millions of tiny files a day, manifests that grow with them, and planning that outlasts the query. Keep the partition at day. city has 60 values and would slice each day into 70 MB pieces. Put it in the sort order instead, and the per-file min and max statistics still let the planner skip files.

The 5-minute micro-batch writes a file per Spark task per commit: 288 commits at 12 tasks leave 3,456 files in each day's partition at around 1.2 MB apiece, and per-request S3 latency on 3,456 objects outweighs the bytes read. Iceberg's rewrite_data_files runs hourly toward a 128 MB target. Name the target: "we would run compaction" without a number sounds like a word heard in a talk.

• 32

About this lesson

This section is part of the Design a Pipeline: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.