Row Groups and Footer Stats
Concepts covered: Predicate Pushdown
Inside a Parquet file, rows are stored in row groups: horizontal slices, each holding a column chunk per column. The footer, written last, lists every row group with a minimum, a maximum and a null count for each column chunk. Given a pushed predicate, the reader fetches the footer once, tests the predicate against each row group's statistics, and reads only the row groups the test cannot rule out.
Reading the statistics
Take a trips file of 8 row groups, with city_id values from 1 to 500. Here are its city_id statistics when rows are written in arrival order and when they are sorted by city_id first.
Run city_id = 42 against each footer. In arrival order every range contains 42, so no row group can be ruled out and the whole file is fetched. Sorted, row group 0 spans 1 to 63 and must be read, while row group 1 starts at 63 and every later one starts higher, so the rest are skipped. Both files hold the same rows and get the same query and the same PushedFilters, so the write order is the only thing that changed the I/O.
- 8 of 8
- 1 of 8
Statistics exist for every column, sorted or not. In both files fare runs from around 2 to about 450 inside every row group, because sorting on city_id scatters fares. So fare > 500 skips every row group and returns nothing without reading a single row group, while fare > 100 skips none. A file can be sorted only one way, so sort it on the column your queries filter on most.
Using the sorted footer above, how many row groups does the reader fetch for city_id = 63?
- Only 1, since city 63 belongs to row group 1 alone
- None, since 63 sits on the boundary between 2 ranges
- All 8, since equality filters cannot skip on int columns
- Exactly 2, since row group 0's max and row group 1's min both equal 63
Writing data that skips well
The cheapest way to cluster is sortWithinPartitions, which sorts each task's rows before they are written and adds no shuffle and no stage:
(trips
.sortWithinPartitions('city_id')
.write
.mode('overwrite')
.parquet('/lake/trips_sorted'))- sortWithinPartitions('city_id')
- No shuffle: each task sorts the rows it already holds
- Each file covers most cities in narrow row-group slices
- With 200 files, city 42 sits in about 200 row groups
- orderBy('city_id')
- Adds a range shuffle and a stage before the write
- Each file covers its own contiguous band of cities
- City 42 sits in 1 or 2 row groups across the table
Pay for the global sort when the table is read far more often than it is written.
Row-group size
Spark's Parquet writer targets row groups of parquet.block.size, 128MB by default, so the 8 row groups above make a 1,024MB file. Override it per write with .option('parquet.block.size', 32 * 1024 * 1024), and the same file holds 32 row groups, so city 42 costs a single 32MB group. The price is a bigger footer, more separate requests to storage and less data per chunk for the codec to compress.
Splits are planned before any footer is read. Spark cuts files at spark.sql.files.maxPartitionBytes, 128MB by default, and a row group goes to the task whose split holds its midpoint. A skipped row group still gets a task; that task finds nothing to fetch and ends almost at once, which shows up as a run of near-zero task durations in the stage.
The seed has no row groups, but you can compute what a footer would record. The cell treats every 25 consecutive transaction_id values as a single row group and reports its row count and the min and max of total_amount. Run it, choose a threshold near the top of the amounts, and decide which groups a reader could skip for total_amount above it. Then change 25 to 100 and watch the ranges widen.
About this lesson
This section is part of the Storage Formats: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.