Inside Parquet

Concepts covered: Reading & Writing

Parquet is the columnar format Spark writes by default. Beyond column blocks, a Parquet file cuts its rows into groups, splits each column into pages, records every column's type, and ends with a map of where everything sits. Each of those pieces exists so that a reader can fetch fewer bytes.

The parts of a file, front to back

PartWhat it holdsDefault size
Magic bytesThe 4 characters `PAR1`, marking the file as Parquet4 bytes
Row groupA horizontal slice of the rows, stored column by columnAbout 128 MB
Column chunkOne column's values for one row groupVaries by column
PageA slice of a column chunk, encoded and compressed as a unitAbout 1 MB
FooterSchema, row counts, and the offset and size of every column chunkUsually KB
Footer length + `PAR1`The footer's size in bytes, then the magic bytes again8 bytes

A file with 2 row groups and 3 columns therefore holds 6 column chunks. The first slice of rows has its own event_id chunk next to the country and dur_ms chunks, and the second slice repeats that set. Row groups exist because a writer buffers a whole row group in memory before writing it out, and because they give readers natural places to split a large file among several tasks.

The footer is read first

A reader opens a Parquet file at the end. The last 8 bytes give the footer's length, 1 more read fetches the footer, and from then on the reader knows the schema and the exact byte range of every column chunk. Here is what the footer of a small events file records for 3 of its columns:

Row groupColumnStarts at byteSize in bytes
0event_id441,200,000
0country41,200,0042,100,000
0dur_ms43,300,00418,700,000
1event_id62,000,00440,900,000
1country102,900,0042,050,000
1dur_ms104,950,00418,400,000

To read country alone, Spark requests 2 byte ranges, starting at byte 41,200,004 and byte 102,900,004: about 4.15 MB from a file of about 123 MB. Without the footer it would have to read the file front to back to find where country begins. The country chunks are small because 2-letter codes repeat, which the encodings below take advantage of.

The schema travels with the data

The footer stores each column's name and type: event_id as a 64-bit integer, country as a string, dur_ms as a 32-bit integer. Spark reads those types instead of guessing them, so a Parquet read never needs the extra pass over the data that inferSchema costs on CSV, and an integer stays an integer from the job that wrote it to every job that reads it.

Encodings: repeats stored once

Before any compression, Parquet encodes each column chunk to exploit how its values repeat. The writer tries dictionary encoding first: it builds 1 dictionary per column chunk, a list of that chunk's distinct values, and stores it as a dictionary page ahead of the chunk's data pages. Every data page then holds small integer codes that point into that single dictionary. If the dictionary grows past its size limit, about 1 MB by default, the writer stops using it and writes the chunk's remaining pages in plain encoding. A high-cardinality column like event_id therefore ends up mostly plain.

The 8 rows become the codes 0 0 1 0 2 0 0 1. In plain form Parquet stores each string as a 4-byte length plus its characters, so BR takes 6 bytes. With 200 distinct countries a code needs 8 bits, so each value drops to 1 byte. Where the same code repeats many times in a row, as when the data is sorted by country, run-length encoding stores the code once with a count: a run of 50,000 BR rows becomes 2 numbers.

What each piece buys a reader

Spark opens a 1 GB Parquet file to read 1 of its 25 columns. What does it read first?

About this lesson

This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.