A Depth Probe
Concepts covered: Spark Execution Model
The walk-through question shows up in Spark interviews under many phrasings, and by now the interviewer assumes you can name the driver, the executors and the cluster manager. The question underneath is about weight: which steps move data, how many bytes, and what that movement does to the wall clock. A list of components answers none of that. The answers that pass this probe attach a cost to each step before the interviewer has to ask for it.
One question, 5 phrasings
| How it is phrased | What it probes | The figure that answers it |
|---|---|---|
| "What happens, step by step, when this job runs?" | Whether you can find the shuffle boundaries in the code | Stages in the plan and tasks per stage |
| "What is the difference between a transformation and an action?" | Whether you know nothing runs until the action, and what the action sets off | Shuffles in the plan the action triggers |
| "This job got slower when the input doubled. Where would you look?" | Whether you know which stage scales with data volume | Shuffle Write per stage |
| "Explain the driver and the executors." | Whether you know what lives where and what crosses the network | Bytes collected to the driver, bytes moved between executors |
| "Why does this join take 40 minutes?" | Whether you check the join strategy before tuning anything else | The smaller side's size against the 10MB threshold |
The way to recognize the probe is the follow-up. If the first answer lists components, the next question is "which of those is expensive?" or "how many tasks would that run?", and you end up producing under pressure what you could have volunteered. If the first answer already places the shuffles and sizes them, the follow-ups become a discussion of tradeoffs, which is where you want the conversation to be.
What the probe listens for
- A correct walk-through
- The driver turns the code into a plan when an action runs
- The plan splits into stages at each shuffle
- Each stage runs a task per partition
- Executors run the tasks and keep shuffle files on local disk
- A walk-through with weight
- Which line of code causes each shuffle, and which stage moves the most bytes
- How many tasks run, from file size and the 200 shuffle default
- Whether the join broadcasts or shuffles both inputs, and the size that decided it
- Which Spark UI metric would confirm or refute each guess
Both columns describe the same machinery. The right one adds 3 things to every claim: a location (the line of code that causes the movement), a size (bytes and task counts, rough ones included) and a check (where in the Spark UI the guess gets confirmed). An answer carrying all 3 lets the interviewer push on any part of it and find something solid underneath.
A worked opener
Suppose the interviewer pastes this job and asks what happens when it runs. The trips table holds March, 500GB of Parquet. The cities table holds a few thousand cities with their boundary polygons, about 6MB on disk.
trips = spark.read.parquet("/data/trips/2024/03/")
cities = spark.read.parquet("/data/dim/cities/")
monthly = (
trips
.filter("status = 'completed'")
.join(cities, "city_id")
.groupBy("city_name")
.agg({"fare": "sum"})
.withColumnRenamed("sum(fare)", "revenue")
.orderBy("revenue", ascending=False)
)
monthly.write.parquet("/data/reports/monthly_revenue/")Count the data movement before naming anything. At about 6MB, cities sits under the 10MB spark.sql.autoBroadcastJoinThreshold, so Spark broadcasts it: the driver collects the table and a copy reaches every executor, while trips never moves for the join. The groupBy adds a hash shuffle on city_name. The orderBy adds a range shuffle on revenue. Two shuffles split the write into 3 stages.
| Stage | What runs in it | Tasks |
|---|---|---|
| 1 | Scan trips, filter, broadcast hash join, partial sum per city | About 4000, a task per 128MB split |
| 2 | Final sum per city, then the write for the range shuffle | 200 planned by the shuffle default. Adaptive execution, on by default since Spark 3.2, coalesces these tiny partitions, so the UI shows far fewer tasks |
| 3 | Sort by revenue, write the report | Up to 200 planned, a task per range, coalesced the same way |
Said aloud, that becomes the opener: "Nothing runs until the write. The driver plans this as a broadcast join and 2 shuffle exchanges, so 3 stages. Stage 1 is the expensive one, about 4000 tasks reading 500GB, but partial aggregation means each task ships at most a row per city into the shuffle. Before tuning anything I'd confirm the broadcast in the SQL / DataFrame tab." Each sentence carries a count or a decision, so the interviewer can follow up on any of them and get a real answer.
The 6MB figure is Spark's size estimate from file statistics. A filter or an explode on the small side can make the real size differ from the estimate, which is another reason the plan check comes before any tuning.
About this lesson
This section is part of the The Architecture Recital lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.