File Format Depth
'Why Parquet?' is a screening question, and the screening answer, 'it is columnar and compressed', earns 1 point of a possible 4. The interviewer wants the mechanisms that let a query read less, because they decide whether a dashboard loads or times out, and because a candidate who knows them also knows when the format is the wrong choice.
Skipping columns, files and row groups
A Parquet file is split into row groups, each holding a slice of rows, and within a row group every column is stored as its own contiguous chunk. Each chunk records statistics, at minimum the smallest and largest value in it, and a footer at the end of the file lists the schema and the byte offset of every chunk. A reader opens the footer first, fetches only the chunks for the columns the query names, and skips any row group whose statistics prove the filter can't match. Those are the 3 skipping mechanisms, and each skips something different.
| Mechanism | What it skips | What it needs from the writer |
|---|---|---|
| Partition pruning | whole directories | the partition column in the WHERE clause |
| Column projection | every column the query does not name | nothing, the layout provides it |
| Row group statistics | groups whose min and max exclude the filter value | data sorted or clustered on the filter column |
Dictionary and run-length encoding often get offered as a fourth mechanism, and the interviewer hears a category error. Encoding skips nothing: a query that needs the status chunk still reads all of it. What encoding changes is the size of that chunk. A dictionary stores each distinct value once and replaces every row's value with a small index, so a status column with 6 possible values needs 3 bits per row, and run-length encoding then collapses a run of identical indexes into 1 value and a count. The 3 skipping mechanisms cut how many chunks are read; encoding cuts the bytes inside each one.
Put figures on it, because the interviewer will. Take a 1 terabyte table of 50 equal-width columns, partitioned by day across 365 days, queried for 1 day and 3 columns. Partition pruning removes the other 364 directories, projection removes 47 columns, and row group statistics on a sorted key remove most of what remains. Each step is a sentence in the answer, and the final figure is the one worth remembering.
| Step | Bytes read |
|---|---|
| Full scan | 1 TB |
| Prune to 1 of 365 daily partitions | about 2.7 GB |
| Project 3 of 50 columns | about 165 MB |
| Statistics on sorted user_id, 1 of about 21 row groups matches | about 8 MB |
The last step assumes row groups of about 128 MB, about 21 per partition; the reader opens 1 group and still fetches only 3 of its 50 column chunks.
Row group statistics only help when values are clustered. If user_id is scattered randomly, every group's minimum sits near 1 and its maximum near the highest id, no group can be ruled out, and the statistics are noise. Sorting or bucketing on the filter column at write time is what turns statistics into skipping, and mentioning the write side separates the answer from the screening version.
When the answer is a row format
- Row formats win (JSON lines, Avro)
- Appending 1 record at a time from a stream
- Fetching 1 whole record by key
- Landing raw payloads before any parsing
- Schemas that change weekly
- Column formats win (Parquet, ORC)
- Scans over millions of rows and a few columns
- Wide tables with 50 or more columns
- Partitions that are written once and read many times
- Output of a compaction job
The common mistake is writing column files directly from a stream. A job that flushes a file every 10 seconds produces 8,640 files per day per partition, each with its own footer, and the query planner opens every footer before reading a byte of data. Past a few thousand files the planning time exceeds the scan time. The standard fix lands stream output in a row format, then compacts it on a schedule into column files sized between 128 megabytes and 1 gigabyte, which keeps row groups large enough for their statistics to mean something.
Schema change is the follow-up tied to this domain. Because the schema is stored in each file's footer, adding a nullable column is safe: old files report the column absent and readers fill null. Renaming a column is worse than an error. Readers match columns by name, old files have no column with the new name, and every row written before the rename reads back as null with nothing failing; a revenue dashboard shows a cliff on the rename date and no alert fires. A type change breaks on old files unless the reader supports that specific widening, such as a 32-bit integer read as a 64-bit one; narrowing, or turning a string into a number, fails. The pipeline stays safe with additive change only, enforced at the write step, with renames handled as a new column plus a view.
About this lesson
This section is part of the How Data Moves: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.