Narrate the Hunt for the Slow Stage
Concepts covered: Spark UI Diagnosis
The Spark UI has 1 screen per level of the hierarchy. The Jobs tab lists every action the program ran, with its duration and the stages it needed. The Stages tab lists every stage across all jobs, with its task count, its duration, its input size, and its shuffle read and shuffle write in bytes. Opening a stage shows the summary metrics for its tasks: min, 25th percentile, median, 75th percentile and max, for task duration, GC time, shuffle read size, and spill to memory and disk. Those 5 columns are the ones you read aloud, because the gap between median and max is the single number that separates the most common causes from each other.
A stage boundary is a shuffle. Spark runs everything it can without moving rows between machines inside one stage, then stops, writes each partition's rows to local disk grouped by destination, and starts the next stage by reading them back over the network. The program below has 1 shuffle, so it runs as 2 stages, and the animation shows which rows move and where they land.
orders.groupBy("region").sum("amount").collect()| order_id | region | amount |
|---|---|---|
| 1 | West | 40 |
| 2 | East | 25 |
| 3 | West | 15 |
| 4 | North | 30 |
| 5 | East | 10 |
| 6 | North | 20 |
| 7 | West | 35 |
| 8 | East | 45 |
| 9 | North | 5 |
Stage 1 reads the 3 input partitions and sums the amounts per region inside each one, so each partition writes at most 3 partial sums, grouped by the task that owns each region. The shuffle routes every partial sum for a region to the same task. Stage 2 adds the partial sums for each region and the driver collects 3 rows. That partial step is why an aggregate's shuffle write can be far smaller than its input. In the Stages tab the job appears as 2 lines, and the second line's shuffle read equals the first line's shuffle write, which is how you confirm that the two lines belong to the same boundary when the list is long.
The hunt, narrated
Here is the Stages tab for the job from the intro, as you would describe it from memory. The job scans a day of viewing events, scans the title catalog, joins them on title_id, aggregates hours watched per title, and writes the result. None of this is visible in the interview room, which is why the narration has to carry every number.
| Stage | What it does | Tasks | Duration | Shuffle write | Shuffle read | Spill (disk) |
|---|---|---|---|---|---|---|
| 0 | Scan viewing_events, prepare join key | 400 | 4 min | 38 GB | 0 | 0 |
| 1 | Scan title_catalog | 8 | 20 s | 1.2 GB | 0 | 0 |
| 2 | Join on title_id, aggregate | 200 | 2 h 51 min | 0.9 GB | 39.2 GB | 24 GB |
| 3 | Write output | 200 | 3 min | 0 | 0.9 GB | 0 |
Stage 2 owns the clock: 2 h 51 min of a 2 h 58 min job. The other 3 stages together take under 8 minutes, so a change to the scan, the catalog read, or the write cannot return more than 8 minutes however well it goes. Say that sentence in the interview. It is the moment the interviewer learns you can tell the 1 stage that matters from the 3 that do not. Then open stage 2 and read its task summary.
| Metric | Min | 25th | Median | 75th | Max |
|---|---|---|---|---|---|
| Duration | 22 s | 34 s | 41 s | 56 s | 2 h 48 min |
| Shuffle read | 60 MB | 110 MB | 135 MB | 160 MB | 12.4 GB |
| Spill (disk) | 0 | 0 | 0 | 0 | 24 GB |
| GC time | 0.4 s | 0.9 s | 1.2 s | 1.6 s | 4 min |
Read the shape. Half the tasks finish in 41 seconds and three quarters in under a minute, yet the stage takes 2 h 51 min because 1 task runs for 2 h 48 min. That task read 12.4 GB where its neighbours read 135 MB, about 90 times more, and it is the only one that spilled. One task holding the stage, with a shuffle read far above the median, is the signature of a hot join key, and the hunt has found both its stage and its task without a screen in the room.
If the stage list is long
A real job may have 30 stages across several actions. Sort the Stages tab by duration and add up the top 2 lines. If they cover 80% of the job, the hunt narrows to those 2. If the time is spread across 20 stages of a few minutes each, say so, because that shape points at a different cause, a slow scan or slow per-row code repeated in every stage, and a different fix than a single hot stage would.
The spoken version, in 6 sentences
- I would open the Jobs tab and find the job holding most of the time.
- In its stages I would sort by duration. Stage 2, the join, has 2 h 51 min of it.
- I would open stage 2 and read the task summary: median 41 s, max 2 h 48 min.
- One task read 12.4 GB of the 39 GB shuffle, and the median task read 135 MB.
- That is a hot value in title_id, so the next check is which value it is.
- After the fix I would re-run on the same input and expect the max task close to the median.
When the interviewer says there is no UI, narrate what each screen would show anyway. The missing screen is the test, and the narration is the answer.
About this lesson
This section is part of the Make It Faster lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.