Inside Parquet
Concepts covered: Reading & Writing
Parquet is the columnar format Spark writes by default. Beyond column blocks, a Parquet file cuts its rows into groups, splits each column into pages, records every column's type, and ends with a map of where everything sits. Each of those pieces exists so that a reader can fetch fewer bytes.
The parts of a file, front to back
| Part | What it holds | Default size |
|---|---|---|
| Magic bytes | The 4 characters `PAR1`, marking the file as Parquet | 4 bytes |
| Row group | A horizontal slice of the rows, stored column by column | About 128 MB |
| Column chunk | One column's values for one row group | Varies by column |
| Page | A slice of a column chunk, encoded and compressed as a unit | About 1 MB |
| Footer | Schema, row counts, and the offset and size of every column chunk | Usually KB |
| Footer length + `PAR1` | The footer's size in bytes, then the magic bytes again | 8 bytes |
A file with 2 row groups and 3 columns therefore holds 6 column chunks. The first slice of rows has its own event_id chunk next to the country and dur_ms chunks, and the second slice repeats that set. Row groups exist because a writer buffers a whole row group in memory before writing it out, and because they give readers natural places to split a large file among several tasks.
The footer is read first
A reader opens a Parquet file at the end. The last 8 bytes give the footer's length, 1 more read fetches the footer, and from then on the reader knows the schema and the exact byte range of every column chunk. Here is what the footer of a small events file records for 3 of its columns:
| Row group | Column | Starts at byte | Size in bytes |
|---|---|---|---|
| 0 | event_id | 4 | 41,200,000 |
| 0 | country | 41,200,004 | 2,100,000 |
| 0 | dur_ms | 43,300,004 | 18,700,000 |
| 1 | event_id | 62,000,004 | 40,900,000 |
| 1 | country | 102,900,004 | 2,050,000 |
| 1 | dur_ms | 104,950,004 | 18,400,000 |
To read country alone, Spark requests 2 byte ranges, starting at byte 41,200,004 and byte 102,900,004: about 4.15 MB from a file of about 123 MB. Without the footer it would have to read the file front to back to find where country begins. The country chunks are small because 2-letter codes repeat, which the encodings below take advantage of.
The schema travels with the data
The footer stores each column's name and type: event_id as a 64-bit integer, country as a string, dur_ms as a 32-bit integer. Spark reads those types instead of guessing them, so a Parquet read never needs the extra pass over the data that inferSchema costs on CSV, and an integer stays an integer from the job that wrote it to every job that reads it.
Encodings: repeats stored once
Before any compression, Parquet encodes each column chunk to exploit how its values repeat. The writer tries dictionary encoding first: it builds 1 dictionary per column chunk, a list of that chunk's distinct values, and stores it as a dictionary page ahead of the chunk's data pages. Every data page then holds small integer codes that point into that single dictionary. If the dictionary grows past its size limit, about 1 MB by default, the writer stops using it and writes the chunk's remaining pages in plain encoding. A high-cardinality column like event_id therefore ends up mostly plain.
The 8 rows become the codes 0 0 1 0 2 0 0 1. In plain form Parquet stores each string as a 4-byte length plus its characters, so BR takes 6 bytes. With 200 distinct countries a code needs 8 bits, so each value drops to 1 byte. Where the same code repeats many times in a row, as when the data is sorted by country, run-length encoding stores the code once with a count: a run of 50,000 BR rows becomes 2 numbers.
What each piece buys a reader
- Footer: Gives the byte range of every column chunk, so the reader fetches only what it needs.
- Column chunks: Keep each column contiguous, so skipping a column skips its bytes.
- Row groups: Let several tasks share a large file, each reading its own row groups.
- Pages: The unit of encoding and compression, small enough to decode in memory.
- Schema: Types arrive with the data, with no inference pass.
Spark opens a 1 GB Parquet file to read 1 of its 25 columns. What does it read first?
- The first row group, to learn the column names from its first page
- The whole file, because column chunks can start anywhere and must be found by scanning
- The magic bytes at the start, which list where each column chunk begins
- The last 8 bytes, which give the footer's length, then the footer with every chunk's offset
About this lesson
This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.