IntermediatePipeline Architecture · 35 min · 2 challenges
The Storage Question: Intermediate
Maya is in the onsite loop at Netflix, the company whose engineers created Apache Iceberg after the directory-based tables in their petabyte-scale Parquet lake kept giving concurrent writers and readers inconsistent answers. The interviewer draws one box labeled 'playback events, 100 billion rows a year' and asks: 'How would you store this?' Parquet, partitions, a table format, a lake or a warehouse, a bill at the end of the month. What the interviewer is testing is whether she can chain them: say why a columnar file reads a fraction of the bytes for a narrow query, pick a partition key by counting its distinct values and its skew, explain what a transaction log gives that a directory listing cannot, decide which workload belongs in a warehouse, and attach a dollar figure to the design. This lesson builds that chain, with the number each link turns on.
What you will be able to do
Explain the 3 mechanisms that make Parquet read fewer bytes than CSV or JSON, with a worked byte count for each
Choose partition keys from a cardinality and skew query, and name the keys that belong in sorting or bucketing instead
Describe the metadata log that gives Delta Lake and Iceberg atomic commits, time travel and schema evolution, and compare the 2 on partition and schema change
Place a raw 500 TB workload and a 2 TB serving workload across lake, warehouse and lakehouse with reasons tied to consumers
Model a year of storage with lifecycle tiers and scan-priced compute, and say which line dominates
Three mechanisms, one file layout
The question arrives in a plain form: the events land as JSON, so why convert them to Parquet before anyone queries them? A complete answer names 3 mechanisms in the file layout and puts a number on each. Columnar storage cuts the bytes a query must read. Column-aware encoding shrinks what remains before compression touches it. Footer statistics let the engine skip whole blocks without reading them.
A CSV or JSON file stores one record after another, so reading the country column means reading every byte of every row. Parquet groups rows into row groups and stores each column's values contiguously inside the group, so a query that touches 3 of 40 columns reads about 3/40 of the bytes, and the file's footer records the byte offset of every column chunk so the engine seeks straight to the chunks it needs.
| Property | CSV | JSON | Parquet |
|---|---|---|---|
| Schema and types | none, every field parsed from text on every read | field names repeated in every record | declared once in the footer, values stored as typed binary |
| Reading one column | full row scan | full row scan | one column chunk per row group |
| Compression | whole-file gzip, not splittable | whole-file gzip, not splittable | per column chunk, after encoding, splittable by row group |
| Statistics | none | none | min, max and null count per column per row group |
| Nested fields | flattened or quoted | native | native, as repeated and nested groups |
Types matter as much as bytes. A CSV timestamp is parsed from text on every read, so a WHERE on event_ts costs a parse per row; a Parquet timestamp is a 64-bit integer compared directly.
Worked example: 1 billion rows, 3 columns
Model an events table of 1 billion rows and 40 columns, covering 1 year of traffic. As JSON with field names in every record, a row runs about 1,200 bytes, so the table is 1.2 TB. As CSV the same row is about 400 bytes, 400 GB in total. Written as Parquet with dictionary encoding and a general compressor, a 6x to 7x reduction over CSV is typical for event data, which puts the table near 60 GB. Now run one query against each copy.
The CSV engine parses all 400 GB because it cannot find the status field without splitting every line. The JSON engine parses 1.2 TB. The Parquet engine reads 3 column chunks per row group: country, event_date and status are low-cardinality strings that dictionary-encode to about 1.5 GB combined across the whole table. Then the footer statistics apply. Each row group records the minimum and maximum event_date it holds, and if the writer produced the files in date order, the single day in the filter lives in a handful of row groups. The table spans 365 days, so that day is about 1/365 of the 1.5 GB: the engine skips the rest and reads around 4 MB.
| Step | CSV | JSON | Parquet |
|---|---|---|---|
| Bytes on disk | 400 GB | 1.2 TB | 60 GB |
| Bytes read for 3 columns | 400 GB | 1.2 TB | 1.5 GB |
| After row group skipping on event_date | 400 GB | 1.2 TB | 4 MB |
| Ratio against Parquet | 100,000x | 300,000x | 1x |
128 MB
row group size most writers default to
2 bits
per value for a 4-value column
Encoding before compression
Compression alone does not explain the 60 GB. Parquet encodes each column chunk according to what its values look like, then compresses the encoded bytes. Take the status column: 1 billion values drawn from 4 distinct strings averaging 7 characters. CSV spends 7 GB on it. Parquet writes a dictionary of the 4 strings once, then stores each value as a fixed-width code of the smallest width that distinguishes 4 entries, and the column drops to 250 MB before any compressor runs. If the rows were sorted by status, run-length encoding records each run as a code and a count, and the same column fits in a few hundred bytes. Integer and timestamp columns get delta encoding, and a general compressor such as zstd finishes the job on bytes that already repeat far less.
Where Parquet loses
Workloads the file layout fights
- ▸Point lookups by id: the engine opens a row group, reads a chunk per column and reassembles the row, which costs more than a row store's single seek.
- ▸One event at a time: Parquet files are immutable, so each append writes a new file, and a day of small files is what the advanced tier of this track is about.
- ▸Updates and deletes: changing one value means rewriting the whole file that holds it.
- ▸Debugging by eye: a column chunk of packed codes tells a human nothing without the footer and the dictionary.
TIP
Say the 3 mechanisms by name, columnar layout, encoding, footer statistics, and attach one number to each: a 3-of-40 column read, a 7 GB column that becomes 250 MB, and a day's row groups found out of a year's. An answer that stops at 'Parquet is compressed' describes gzip, which CSV has too.
What a partition physically is
A partition is a directory. A table partitioned by event_date stores the 20 October rows under events/event_date=2026-10-20/ and the next day's rows under the next directory, with the partition column absent from the files themselves because its value is the directory name. When a query filters on a constant, the engine matches it against directory names and lists only the directories that match. That pruning happens before any file is opened, which places it one level above the row group statistics in the previous section: partitions decide which directories to list, statistics decide which row groups inside those files to read.
Cardinality analysis
The partition count is the product of the distinct values of every partition key, and each partition should hold enough data to fill files of 128 MB to 1 GB. Those 2 facts turn the question into arithmetic. Before choosing, measure the candidates against the data you have.
Take the events table at 50 GB of Parquet per day and score each candidate key set by partitions per year, average partition size, and skew between the largest and smallest partition.
| Partition keys | Partitions per year | Average size | Largest to smallest | Verdict |
|---|---|---|---|---|
| event_date | 365 | 50 GB | 1.5x, weekends are lighter | fits any query that filters by day |
| event_date, hour | 8,760 | 2.1 GB | 6x, nights are quiet | fits only if queries filter by hour |
| event_date, event_type (30) | 10,950 | 1.7 GB | 40x | workable if most queries name a type |
| event_date, country (200) | 73,000 | 250 MB | 5,000,000x, 10 GB against 2 KB | most partitions are a single tiny file |
| event_date, user_id (50 M) | up to 18 billion | 1 KB | none, every partition is tiny | never |
The over-partition trap
Over-partitioning costs in 3 places. Listing: on object storage each directory is a list call that takes tens of milliseconds, so a query spanning 73,000 country partitions spends minutes planning before it reads a byte. Small files: every writer that touches a partition produces at least 1 file there, so 24 hourly loads into date-country partitions produce 24 files of about 400 MB in the largest country, 24 files near 10 MB in a mid-sized one, and 24 files of a few bytes in the smallest, and each file costs a footer read and a request. Statistics: when a partition holds 1 row group, min and max cover the whole partition and skip nothing. The symptom in the interview scenario is a dashboard showing a 30-day trend across all countries: it touches 6,000 directories and 144,000 files, and spends minutes on list calls and footer reads before it scans any data.
50 ms
per directory list on object storage
High-cardinality keys: sort or bucket, never partition
Queries that filter on user_id still need help, and the help lives inside the partition. Sorting each day's files by user_id makes every row group hold a narrow range of ids, so the footer min and max prune most row groups for a single-user lookup: with 50 GB in 400 row groups per day, a lookup touches 1 or 2 of them. Bucketing hashes user_id into a fixed number of files per partition, which gives joins between 2 tables bucketed the same way a file-to-file match with no shuffle, and gives lookups a single file to open.
✓Partition on it when
- Queries filter on it with constants most of the time
- Distinct values stay in the hundreds or low thousands
- Each value holds at least 1 GB over the table's life
- Old values stop receiving writes, so a date or a month
✗Sort or bucket on it when
- Distinct values run to millions, such as user or order ids
- It is the join key between 2 large tables
- Lookups need one file rather than one directory
- The value is spread across all time, so no partition would ever close
TIP
Answer in this order: which columns appear as constants in the top queries, what the cardinality and skew query shows for each, what size each partition reaches, and how the keys you rejected get handled by sorting or bucketing. Naming the rejected keys is what separates a partition choice from a partition guess.
The problem a directory cannot solve
A partitioned directory of Parquet files has no definition of the table other than whatever a list call returns at the moment it runs. That produces 4 failures the interviewer will probe. A writer that dies after uploading 3 of 8 files leaves 3 orphans that every reader counts. Two writers that overwrite the same day each believe they finished, and the last upload wins silently. A reader that starts during a rewrite sees old files for one partition and new files for another. And the schema is whichever footer the engine opens first, so a column added last week reads as null or as an error depending on file order.
Both Delta Lake and Iceberg answer with the same idea: a log, stored next to the data, that records which exact files make up each version of the table. Readers consult the log and never list the directory. Writers add files, then commit by writing one new log entry, and the commit either lands whole or fails. Everything the interviewer calls ACID, time travel and schema evolution follows from where that log lives and what it records.
How each format writes the log
Delta Lake keeps a _delta_log/ directory of numbered JSON commits, 00000000000000000041.json for commit 41, each listing the files added and removed and any schema change. Every 10 commits it writes a Parquet checkpoint of the full state so readers replay only the tail. A commit is the atomic creation of the next number; if 2 writers race for 41, the object store accepts 1 and the other rereads the log and retries as 42.
Iceberg keeps a tree. A metadata.json file names the current snapshot and the schema and partition spec history; each snapshot points to a manifest list; each manifest lists data files with per-file column statistics. A commit swaps the pointer to the new metadata file in the catalog, 1 atomic operation, and a loser in a race rebases on the winner's snapshot and retries. Because statistics live in the manifests, planning a query reads a few metadata files rather than 73,000 directory listings.
Three axes, compared
| Property | Delta Lake | Iceberg |
|---|---|---|
| Atomic commit | create the next numbered log file; collisions retry | swap the catalog pointer to a new metadata file; collisions rebase and retry |
| Isolation | readers pin to the log version they started on | readers pin to the snapshot they started on |
| Time travel | by version number or timestamp, back to the oldest retained log entry | by snapshot id or timestamp, back to the oldest retained snapshot |
| Retention | VACUUM removes files older than the retention window, 7 days by default | expire_snapshots removes snapshots and the files only they reference |
| Schema evolution | add, rename and drop; rename and drop require column mapping mode so files are matched by id rather than name | columns carry ids from creation, so add, rename, drop and reorder never touch data files |
| Partition evolution | partition columns are physical; changing them rewrites the table | partition spec is versioned; new data uses the new spec, old files keep theirs, queries plan across both |
| File statistics | per file in the log, first 32 columns by default | per file in manifests, configurable per column |
Making the choice in the room
The interviewer wants a decision tied to properties, and the properties come from the workload. Three questions settle it. How many engines read and write the table? Pick the format every engine in the stack already supports natively, because a format one engine writes and another cannot read is a copy job waiting to happen. Will the partition scheme change? A table that starts partitioned by day and needs hours 6 months later rewrites 9 TB under Delta Lake and changes a spec under Iceberg. How often does the schema change by rename? Iceberg's column ids make it safe by default; Delta Lake needs column mapping enabled before the first rename.
Then name the cost both formats share, because the interviewer will ask. Every rewrite keeps the old files until retention expires them. The events table rewrites 3 days of late-arriving partitions each morning, 150 GB of new files a day, and with a 30-day retention window the old versions hold 4.5 TB that exist for time travel alone. That number reappears on the bill in the last section.
TIP
Describe the log before naming a format: 'a commit is one atomic write of a metadata entry that names the files in this version'. Then answer Delta or Iceberg with the 3 workload questions. An answer that opens with a brand and no mechanism gets the follow-up 'and what does that do for concurrent writers?', which the mechanism already answered.
Three shapes, separated by who owns the files
A warehouse stores tables in its own internal format that only its engine reads, and in return gives transactions, row-level updates, indexes or clustering, and query performance that nobody has to tune at the file level. A data lake stores open files, Parquet or JSON, on object storage that any engine can read, charges the object store's rate for storage, and gives no transactions, no schema enforcement and no consistent view unless something adds them. A lakehouse is a lake with a table format from the previous section and a catalog on top, which restores the transactional properties on open files, so the engine choice stays open while the correctness returns.
| Axis | Warehouse | Data lake | Lakehouse |
|---|---|---|---|
| Who can read the files | the warehouse engine | any engine with a Parquet reader | any engine that speaks the table format |
| Transactions | full, row level | none without a table format | snapshot-level ACID from the log |
| Updates and deletes | in place, cheap | rewrite files by hand | rewrite files, tracked by the log |
| Raw and semi-structured data | loaded and typed first | stored as landed | stored as landed, promoted when typed |
| Storage price | the vendor's rate, often near object storage | object storage rate | object storage rate plus log and snapshot overhead |
| Compute price | billed by the vendor's unit on every read | whichever engine you run, including free ones | same as the lake |
| Operational burden | lowest | highest: compaction, statistics and consistency are yours | medium: the format handles consistency, compaction stays yours |
A scenario with 2 workloads
Place a company with 2 very different tables. The first is 500 TB of raw clickstream, retained 3 years, read mostly by model training jobs that scan months of Parquet, with an analyst querying it a few times a week. The second is 2 TB of curated finance marts, read by 400 dashboard users who expect sub-second filters and by an audit process that needs the numbers as they stood on the last day of each quarter. Loading all 500 TB into the warehouse means paying its storage rate on 500 TB and its compute on every training scan, with the training framework reading through a connector instead of opening files. Keeping the finance marts as bare Parquet in the lake means 400 users hitting files with no indexes and an audit that cannot reconstruct a quarter end.
The design that fits puts the clickstream in the lake as raw JSON, promotes it to Parquet under a table format for the curated layer, and serves the finance marts from the warehouse or from a gold lakehouse layer with time travel enabled, depending on which engine the dashboard tool already uses. The layers also answer the retry and late-data probes: the curated layer overwrites a partition per run, so a retried run produces the same partition, and a late event lands in the partition of its event date and triggers a rewrite of that day.
Sources
Raw
Curated
Serving
Consumers
JSON
as emitted
nightly extract
object storage
as landedSLA15 minRETRY3IDEMPOTENCYfile name = batch id
Parquet + table format
SLAhourlyBACKFILLrewrite by event dateIDEMPOTENCYoverwrite partition
warehouse or gold table
SLAdaily 06:00MONITORrow count vs prior day
scans months of Parquet
400 users
Raw stays open and cheap, the curated layer gains transactions from the table format, and only the 2 TB that 400 people query sub-second pays warehouse prices.
Where the lakehouse pays off
The lakehouse earns its place when the same 500 TB has to serve both audiences: the training jobs keep opening Parquet directly, and an analyst gets a consistent snapshot, a schema enforced on write and a quarter-end version to query, without a second copy in the warehouse. The price is operational: compaction schedules, statistics collection and retention windows are the team's job, where a warehouse would run them unseen. A design that chooses lakehouse should name who owns those 3 jobs.
The questions that decide it
Who reads it, how often, and with what latency expectation: 400 users at sub-second or 3 jobs at hours
How big it is in TB, because storage rate differences multiply by that figure
Whether rows mutate after landing or only append, since updates on bare files mean rewrites
How many engines need the same table, since a proprietary format stops at 1
Whether an auditor needs the table as it stood on a date, which requires a log or a warehouse
Who will run compaction and statistics if the files are yours
Answers that lose the thread
- ▸'Lake, because storage is cheaper' with no consumer named: the 400 dashboard users have no index and no transactions.
- ▸'Warehouse for everything': 500 TB of training data paid for twice, storage and per-scan compute, through a connector.
- ▸'Lakehouse' as a brand word: the interviewer asks what the log does, and the mechanism from the previous section is the answer they want.
The lines on the bill
A storage estimate has 4 lines, and in analytics the last one usually dominates. Storage is billed per GB-month by tier. Requests are billed per 1,000 calls, with list and put costing about 10 times a get. Retrieval from cold tiers is billed per GB restored, on top of a wait. Compute is billed either by the TB a query scans or by the hours a cluster runs, and the scan model prices the format and partition decisions from the first 2 sections directly.
| Tier | Price per GB-month | Retrieval | Minimum duration | Fits |
|---|---|---|---|---|
| Standard | $0.023 | none, milliseconds | none | last 30 days, queried daily |
| Infrequent access | $0.0125 | $0.01 per GB, milliseconds | 30 days | 31 to 365 days, queried weekly |
| Archive | $0.0036 | per GB plus hours to restore | 90 days | retained for compliance, read once a year or never |
Prices are the published list prices of the largest cloud object store for its US regions in 2026, rounded. Your provider and region differ by tens of percent, and the ratios between tiers are what the model turns on.
Worked model: 1 year of the events table
Scale the events table to the 50 GB of Parquet a day the partition section assumed, which is about 330 GB a day as CSV at the first section's 400 GB to 60 GB ratio. After 1 year the Parquet copy holds 18.25 TB and the CSV copy about 120 TB. Held entirely in the standard tier, month 12 costs 18,250 GB times $0.023, which is $420 for Parquet, and 120,450 GB times $0.023, which is $2,770 for CSV. The format decision alone moved $2,350 a month before a single query ran.
Now apply a lifecycle policy to the Parquet copy: standard for 30 days, infrequent access until day 90, archive after. The 1,500 GB in standard costs $34.50. The 3,000 GB in infrequent access costs $37.50. The 13,750 GB in archive costs $49.50. The month-12 bill drops from $420 to $121.50. The catch comes from the third section's consumer question: a dashboard that compares this quarter to the same quarter last year now touches archived files, which cannot be read in place. Its query fails on them until someone issues a restore, which takes hours, bills per GB restored and produces a temporary copy. That is why the archive transition belongs after the longest query window: if analysts query 13 months back, the archive step moves to day 400, and the bill becomes $34.50 plus 16,750 GB times $0.0125, which is $243.88.
Compute: where the format decision gets multiplied
Engines that bill per TB scanned commonly list around $5 per TB. A query that reads the full year of CSV scans about 120 TB and costs about $600. The same query on Parquet without a partition filter scans 18.25 TB and costs $91. Reading only the 3 columns it needs, which hold the same 2.5% of the bytes they did in the first section, it scans about 450 GB and costs about $2.30. Adding the date partition filter cuts that to one day of those columns, about 1.25 GB, for about 0.6 cents. Multiply by the 1,000 queries a day a 400-user dashboard produces and the storage figures above stop mattering: $600,000 a day against $91,000 against $2,300 against $6.25. The interviewer who asks 'how much will this cost' is usually checking whether the candidate reaches that multiplication.
1.8x
standard over infrequent access, per GB
6.4x
standard over archive, per GB
6.7x
CSV over Parquet, bytes scanned per year
Two smaller lines that bite
Request costs rarely reach dollars but reveal the small-file problem. If the hourly loads from the partition section produced 1 million files over the year, a full scan issues 1 million get requests at about $0.0004 per 1,000, which is $0.40, and spends minutes on footer reads; the money is negligible and the latency is the whole story, which the advanced tier takes up. Time travel from the table format section is a storage line: the 4.5 TB of expired-but-retained versions sits in the standard tier at $103.50 a month, so a retention window is a budget decision as much as a recovery one.
✓Do
- Convert at landing so the 6.7x size ratio applies to every downstream copy
- Set lifecycle tiers from the longest query window rather than the ingest rate
- Price a query by bytes scanned, and quote it per day at dashboard volumes
- Count retained snapshots as storage you pay for
✗Don't
- Quote storage alone when a per-scan engine is in the design
- Archive data a dashboard still reads
- Mistake request dollars for the small-file cost; the cost is latency
- Assume list prices match across providers and regions
TIP
Open with the line that dominates: 'at $5 per TB scanned, the format and the partition key set the bill, and storage is $420 a month at most'. Then give the lifecycle numbers. A cost answer that starts with the GB-month price reads as a reseller's, and one that starts with bytes scanned reads as an engineer's.
❯❯❯PUTTING IT ALL TOGETHER
> A food delivery company lands 60 GB a day of order and courier location events as JSON, must keep them 5 years for disputes, trains delivery-time models on 6 months of history, settles late order corrections every morning, and serves a finance dashboard of settled orders to 300 people who compare each month to the same month last year.
Converting the 60 GB of JSON to Parquet at landing cuts the stored footprint by roughly 20x, to about 3 GB a day, and lets the model training jobs read only the columns they use.
Partitioning by order_date gives 365 partitions a year of about 3 GB each, while courier_id goes into bucketing so the location join avoids a shuffle.
The morning settlement rewrites 3 days of partitions, so a table format's atomic commit keeps the dashboard from seeing a half-written day, and its retention window holds 30 days of old versions at a storage cost worth stating.
Location events stay in the lake for the training jobs, settled orders move to the warehouse or a gold table with time travel because 300 people need indexed, audited numbers.
The 13-month comparison in the dashboard fixes the lifecycle policy: infrequent access after 30 days, archive only after 400, and the per-scan bill for 300 users is the number to say first.
KEY TAKEAWAYS
Parquet wins on 3 mechanisms, columnar chunks, encoding before compression, and footer statistics; a 3-column query on 1 billion rows reads 4 MB instead of 400 GB.
A partition is a directory, its count is the product of key cardinalities, and each should reach 1 GB; high-cardinality keys belong in sorting or bucketing.
Delta Lake and Iceberg both replace directory listing with a log that names each version's files; they differ most on schema ids and partition evolution.
Lake, warehouse and lakehouse are decided by consumers, volume, mutation pattern and engine count, with the log supplying what bare files lack.
At $5 per TB scanned, compute dominates storage, so format and partition key set the bill; lifecycle tiers follow the longest query window.
Parquet, partitions, table formats, lake or warehouse, and the bill: the storage question with its numbers
- Category
- Pipeline Architecture
- Difficulty
- intermediate
- Duration
- 35 minutes
- Challenges
- 2 hands-on challenges
Topics covered: Why Parquet?, How Would You Partition?, Delta or Iceberg?, Data Lake or Warehouse?, How Much Will This Cost?
Lesson Sections
- Why Parquet? (concepts: Columnar vs Row Formats)
Three mechanisms, one file layout The question arrives in a plain form: the events land as JSON, so why convert them to Parquet before anyone queries them? A complete answer names 3 mechanisms in the file layout and puts a number on each. Columnar storage cuts the bytes a query must read. Column-aware encoding shrinks what remains before compression touches it. Footer statistics let the engine skip whole blocks without reading them. Types matter as much as bytes. A CSV timestamp is parsed from t
- How Would You Partition? (concepts: Partitioning Strategies)
What a partition physically is Cardinality analysis The partition count is the product of the distinct values of every partition key, and each partition should hold enough data to fill files of 128 MB to 1 GB. Those 2 facts turn the question into arithmetic. Before choosing, measure the candidates against the data you have. Take the events table at 50 GB of Parquet per day and score each candidate key set by partitions per year, average partition size, and skew between the largest and smallest p
- Delta or Iceberg? (concepts: Table Formats)
The problem a directory cannot solve A partitioned directory of Parquet files has no definition of the table other than whatever a list call returns at the moment it runs. That produces 4 failures the interviewer will probe. A writer that dies after uploading 3 of 8 files leaves 3 orphans that every reader counts. Two writers that overwrite the same day each believe they finished, and the last upload wins silently. A reader that starts during a rewrite sees old files for one partition and new fi
- Data Lake or Warehouse? (concepts: Data Lake Architecture)
Three shapes, separated by who owns the files A warehouse stores tables in its own internal format that only its engine reads, and in return gives transactions, row-level updates, indexes or clustering, and query performance that nobody has to tune at the file level. A data lake stores open files, Parquet or JSON, on object storage that any engine can read, charges the object store's rate for storage, and gives no transactions, no schema enforcement and no consistent view unless something adds t
- How Much Will This Cost? (concepts: Columnar vs Row Formats, Compression & Encoding)
The lines on the bill A storage estimate has 4 lines, and in analytics the last one usually dominates. Storage is billed per GB-month by tier. Requests are billed per 1,000 calls, with list and put costing about 10 times a get. Retrieval from cold tiers is billed per GB restored, on top of a wait. Compute is billed either by the TB a query scans or by the hours a cluster runs, and the scan model prices the format and partition decisions from the first 2 sections directly. Prices are the publishe