Push It Down

A medium Spark interview practice problem on DataDriven. Write the Spark code and run it against the tables, with instant feedback.

Domain

Spark

Difficulty

medium

Seniority

L5

Asked in

TikTok

Problem

user_events is stored as one folder per event_date, and the growth team's 2026 activity rollup has been reading every year of history because its date bound was computed in a separate lookup the file scan never sees. Rebuild the rollup with the bound written straight against event_date: one row per country with the number of events since the start of 2026.

You renamed the column. Catalyst forgot how to prune.

Expected output

Worked solution and explanation

What this is really testing

This is a plan-reading question dressed up as a slow daily job. The logic is a one-line filter and a count; nobody fails on that. What separates candidates is knowing that a correct filter and a pushed filter are different things. Catalyst can only skip date folders when the bound on event_date is a plain predicate it can evaluate while planning the scan. Hide the bound behind a scalar subquery and the plan shows PartitionFilters: []: every executor opens all 1,096 folders, reads 3 TB, and throws 99.7% of it away after the fact.

The shape that defeats pruning

events = spark.table("user_events")
bound = spark.sql("SELECT DATE '2026-01-01' AS start_day")

# Joining in the bound hides it from the scan planner
this_year = events.crossJoin(bound).filter(F.col("event_date") >= F.col("start_day"))

result = this_year.groupBy("country").agg(F.count("*").alias("events"))
result

The answer this returns is right, which is exactly why it ships. Partition pruning happens before any task launches: the driver lists folders and keeps only those whose event_date value satisfies the partition filters. A value that comes from another relation or a subquery is not known at that moment, so the predicate stays as a post-scan Filter node and the listing keeps everything.

Step 1: Read the FileScan line first

Run explain() and go straight to the scan. PartitionFilters: [] with a Filter sitting above FileScan means the date bound never reached storage. The stage input size in the Spark UI tells the same story: terabytes where you expected gigabytes.

Step 2: Put the bound directly on event_date

Write the comparison against the raw partition column with a value Catalyst can fold at plan time: a literal, or a deterministic expression like F.date_sub(F.current_date(), 1). No rename first, no join, no subquery between the column and the bound.

Step 3: Count after the scan is already small

With the folders pruned, the groupBy on country shuffles only the surviving rows. The partial aggregate runs per task before the exchange, so the shuffle carries one row per country per task, not one per event.

events = spark.table("user_events")

# A plain comparison on the partition column is visible to the scan planner,
# so it lands in PartitionFilters and whole date folders are skipped.
this_year = events.filter(F.col("event_date") >= "2026-01-01")

result = this_year.groupBy("country").agg(F.count("*").alias("events"))
result

Post-scan filter

Bound comes from a subquery or join; PartitionFilters: []; every folder is listed and read; the Filter drops rows after I/O is paid.

Partition filter

Bound is a literal or foldable expression on event_date; the driver prunes the folder list; tasks read only matching files.

Common follow-up questions

• The bound must come from a control table such as the last successful load date. How do you keep pruning? (Collect the value to the driver first, or rely on dynamic partition pruning and explain when it applies.)
• How would you confirm pruning without reading the plan? (Stage input size and files read in the Spark UI.)
• What happens if `event_date` is also stored inside the Parquet files, not only in the folder name? (Data filters versus partition filters, and Parquet row-group statistics.)