Partition Pruning
Concepts covered: Predicate Pushdown
When a filter references only partition columns, Spark tests it against directory names before it opens a single file. On events, written with partitionBy("event_date") and holding 730 days, a query for 1 day rules out 729 directories from their names alone. Their files are never opened or scanned. That is partition pruning, the return on the layout work of the previous section.
- 29.2TB
- 40GB
Seeing the prune in the plan
events = spark.read.parquet("/lake/events")
one_day = events.filter(
(F.col("event_date") == "2024-03-15") & (F.col("country") == "US")
)
one_day.explain()explain() prints a scan node that sorts the filters by where they apply. These are the parts of it to read:
| Plan text | What it means |
|---|---|
| PartitionFilters: [isnotnull(event_date#3), (event_date#3 = 2024-03-15)] | Checked against directory names. Only matching directories are read. |
| DataFilters: [isnotnull(country#2), (country#2 = US)] | Checked against rows inside the files that survive pruning. |
| PartitionFilters: [] | Nothing to prune with. Every directory under the path is read. |
| Location: InMemoryFileIndex(1 paths)[file:/lake/events] | The root Spark lists from. Pruning happens beneath it. |
The scan also prints a PushedFilters list. It belongs to a separate mechanism inside the file format and does not affect which directories are read.
The plan shows intent. To confirm the run, open the query in the Spark UI after it finishes: the scan node reports the number of files read, which should match the file count of the 1 matching directory, a few hundred here, instead of the whole table's.
Which predicates prune
Spark classes a filter as a partition filter when every column it references is a partition column and the expression is deterministic. It evaluates that expression on the values parsed from directory names, so a function of the partition column prunes as well as a plain comparison does. A filter that also touches a data column stays at row level.
- Prunes on event_date
- event_date = '2024-03-15'
- event_date >= '2024-03-01' AND event_date < '2024-04-01'
- event_date IN ('2024-03-14', '2024-03-15')
- month(event_date) = 3
- event_date = '2024-03-15' AND country = 'US', where the date half prunes
- Reads every directory
- event_ts >= '2024-03-15 00:00:00', a data column
- event_date = '2024-03-15' OR country = 'US'
- country = 'US' on its own
The OR row catches teams out. A row with country = 'US' could sit under any date, so Spark cannot rule a directory out. If the date branch needs to be fast, run it as a query of its own.
Two ways to lose the prune
Filtering on the column the partition was derived from skips pruning. If event_date was computed from event_ts at write time, a dashboard filtering event_ts to a 6-hour window still reads all 730 directories, because Spark has no way to know the 2 columns agree. Adding the matching event_date predicate leaves the result unchanged and cuts the read to 1 directory:
window = events.filter(
(F.col("event_ts") >= "2024-03-15 06:00:00")
& (F.col("event_ts") < "2024-03-15 12:00:00")
& (F.col("event_date") == "2024-03-15")
)Pointing the read straight at a directory is the other way to lose it. spark.read.parquet("/lake/events/event_date=2024-03-15") reads the right files but treats that directory as the root, so event_date disappears from the schema. Adding .option("basePath", "/lake/events") keeps the column. Reading the root and filtering reaches the same files and keeps it too.
The events table is written with partitionBy("event_date") and also stores event_ts and country. Which filter reads only the directories for March 2024?
- F.year("event_date") == 2024 combined with F.month("event_date") == 3
- F.col("event_ts").between("2024-03-01", "2024-03-31 23:59:59")
- (F.col("event_date") >= "2024-03-01") | (F.col("country") == "US")
- F.col("event_ts").cast("date") >= "2024-03-01" with a matching upper bound
About this lesson
This section is part of the Partitioning: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.