Why Parquet?

Concepts covered: Columnar vs Row Formats

Three mechanisms, one file layout

The question arrives in a plain form: the events land as JSON, so why convert them to Parquet before anyone queries them? A complete answer names 3 mechanisms in the file layout and puts a number on each. Columnar storage cuts the bytes a query must read. Column-aware encoding shrinks what remains before compression touches it. Footer statistics let the engine skip whole blocks without reading them.

A CSV or JSON file stores one record after another, so reading the country column means reading every byte of every row. Parquet groups rows into row groups and stores each column's values contiguously inside the group, so a query that touches 3 of 40 columns reads about 3/40 of the bytes, and the file's footer records the byte offset of every column chunk so the engine seeks straight to the chunks it needs.

Types matter as much as bytes. A CSV timestamp is parsed from text on every read, so a WHERE on event_ts costs a parse per row; a Parquet timestamp is a 64-bit integer compared directly.

Worked example: 1 billion rows, 3 columns

Model an events table of 1 billion rows and 40 columns, covering 1 year of traffic. As JSON with field names in every record, a row runs about 1,200 bytes, so the table is 1.2 TB. As CSV the same row is about 400 bytes, 400 GB in total. Written as Parquet with dictionary encoding and a general compressor, a 6x to 7x reduction over CSV is typical for event data, which puts the table near 60 GB. Now run one query against each copy.

SELECT
  country,
  COUNT(*) AS error_events
FROM events
WHERE event_date = DATE '2026-10-20'
  AND status = 'error'
GROUP BY country

The CSV engine parses all 400 GB because it cannot find the status field without splitting every line. The JSON engine parses 1.2 TB. The Parquet engine reads 3 column chunks per row group: country, event_date and status are low-cardinality strings that dictionary-encode to about 1.5 GB combined across the whole table. Then the footer statistics apply. Each row group records the minimum and maximum event_date it holds, and if the writer produced the files in date order, the single day in the filter lives in a handful of row groups. The table spans 365 days, so that day is about 1/365 of the 1.5 GB: the engine skips the rest and reads around 4 MB.

• 128 MB
• 2 bits

Encoding before compression

Compression alone does not explain the 60 GB. Parquet encodes each column chunk according to what its values look like, then compresses the encoded bytes. Take the status column: 1 billion values drawn from 4 distinct strings averaging 7 characters. CSV spends 7 GB on it. Parquet writes a dictionary of the 4 strings once, then stores each value as a fixed-width code of the smallest width that distinguishes 4 entries, and the column drops to 250 MB before any compressor runs. If the rows were sorted by status, run-length encoding records each run as a code and a count, and the same column fits in a few hundred bytes. Integer and timestamp columns get delta encoding, and a general compressor such as zstd finishes the job on bytes that already repeat far less.

Where Parquet loses

About this lesson

This section is part of the The Storage Question: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.