Column Projection
Concepts covered: Predicate Pushdown
Projection is the list of columns a query keeps. With a Parquet source, Spark hands that list to the file reader, which turns it into byte-range requests for the matching column chunks. Columns left out of the list are never fetched or decoded, and they take no executor memory.
Catalyst writes the column list for you
The job below never calls select. It filters on event_type first, then groups on country and sums dur_ms. When Catalyst rewrites the plan, it traces which columns each step uses and moves that list into the scan, so only 3 of the 30 columns are read.
events = spark.read.parquet("/data/events/")
report = (
events
.filter(events.event_type == "purchase")
.groupBy("country")
.sum("dur_ms")
)
report.collect()Run report.explain() and find the FileScan parquet line of the plan. Its ReadSchema entry is the column list the reader received: struct<country:string,event_type:string,dur_ms:int>. When a Parquet scan is slower than its column list suggests, this line is the first thing to check: a ReadSchema listing all 30 columns means something in the job asked for them.
What the narrower read saves downstream
Every later step works on the narrower rows. The groupBy above shuffles rows that carry only country and dur_ms, because the filter has already used up event_type before the shuffle. If one of the 30 columns is a long free-text user_agent, it stays out of the plan and never crosses the network.
Four ways a job loses its projection
Catalyst trims only the columns it can prove are unused. These patterns hide that information or ask for everything:
- select("*") or collecting a DataFrame that still has every column: the plan really does need all 30.
- events.cache() before narrowing: the cache stores what it was given, so the scan reads all 30 columns to fill it.
- events.rdd.map(...): the lambda is opaque to Catalyst, so every column stays in case the function reads it.
- A CSV or JSON source: Spark can skip converting unused fields, but every byte of every line still comes off storage.
slim = (
spark.read.parquet("/data/events/")
.select("country", "event_type", "dur_ms")
.cache()
)
slim.filter(slim.event_type == "purchase").groupBy("country").sum("dur_ms").collect()- Do
- Select the columns a cached or written DataFrame needs before cache() or write
- Check ReadSchema in explain() when a Parquet scan reads more than you expect
- Stay in DataFrame operations so Catalyst can see which columns each step uses
- Avoid
- Cache a wide source table whole and narrow it in later steps
- Drop to .rdd for logic that DataFrame functions can express
- Expect a select on a CSV source to cut the bytes read from storage
Try it
The cell reads the practice table page_views, which has 7 columns, and averages dur_ms per device. It has no select, so the column list comes from Catalyst alone. Before you run it, decide which columns its scan should fetch. Then change .avg("dur_ms") to .agg({"dur_ms": "avg", "view_id": "count"}) and work out how that list should change before the second run.
On a Parquet source, the format Spark writes when none is named, the first version's explain() shows a ReadSchema holding dur_ms and device: 2 of the 7 columns, with no select written anywhere. The second version adds view_id to that list, because the count now uses it, and each returned row gains a count beside its average. The column list follows what the plan touches. It holds only while every step is one Catalyst can read, and the next question shows what happens when a step is not.
The events table has 30 columns. How many of them does the Parquet scan in this program read?
events = spark.read.parquet("/data/events/")
pairs = events.rdd.map(lambda r: (r.country, r.dur_ms)).collect()- All 30, because Catalyst cannot see which fields the lambda reads and keeps every column
- 2, because Catalyst reads the lambda and keeps only country and dur_ms
- 2, because collect() returns only the fields the lambda puts in its tuples
- 0, because .rdd reads from the cache and never from the Parquet files
About this lesson
This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.