The Five Checks, in Order
Slow-job investigations tend to start with whichever number looks most alarming. An executor shows red GC time, so someone doubles spark.executor.memory. The cluster bill rises and the ranking job still runs 2.3 hours, because the cause was a single title_id holding 35 times the typical task's rows. The memory change treated a symptom of that key.
A fixed order prevents that. Each check reads one number from one place in the UI, and each narrows where the next looks.
- 1. Slow stage (transform, Stages tab)
- 2. Shuffle volume (transform, Stage page header)
- 3. Skew (transform, Summary Metrics)
- 4. GC time (transform, Executors tab)
- 5. Spill (transform, Stage page columns)
- 1. Slow stage -> 2. Shuffle volume
- 2. Shuffle volume -> 3. Skew
- 3. Skew -> 4. GC time
- 4. GC time -> 5. Spill
| Check | Where you read it | The number | Reads as healthy |
|---|---|---|---|
| 1. Slow stage | Stages tab, sorted by Duration | Stage duration as a share of the job | Picks the target for the other 4 checks |
| 2. Shuffle volume | Stage page header | Shuffle write over input | Matches the operation's shape |
| 3. Skew | Summary Metrics, Duration row | Max task over median task | Under 2 |
| 4. GC time | Executors tab, Task Time (GC Time) | GC time over task time | Under 10% |
| 5. Spill | Stage page, Spill (Memory) and Spill (Disk) | Any spill at all | Columns absent |
Checks 2, 3 and 5 live on the stage detail page, which opens when you click a stage's description in the Stages tab. Check 4 has a tab of its own because the JVM records garbage collection per executor; the stage page also carries a GC Time row, useful once you know which executor to blame.
Why this order
The order runs from the widest view to the narrowest. Shuffle volume is a stage total: if a stage moves 5 times the bytes its operation needs, every task inside it is oversized, and its skew, GC and spill readings all describe that excess. Skew comes next because it compares tasks with each other, and a task holding far more rows than its peers is also the one collecting garbage and spilling. GC goes before spill because the pair separates 2 causes: spill means sort and aggregation buffers ran out of execution memory, while high GC with zero spill points at something else filling the heap, usually cached data.
Running the checks on the ranking job
Here is the Stages tab for the title-ranking job after its late run, 2.3 hours end to end:
| Stage | Description | Duration | Tasks | Input | Output | Shuffle Read | Shuffle Write |
|---|---|---|---|---|---|---|---|
| 3 | scan viewing_events | 7.4 min | 2,800 | 380 GB | 0 B | 0 B | 205 GB |
| 4 | window by title_id, write | 2.1 h | 1,000 | 0 B | 1.6 GB | 205 GB | 0 B |
Check 1 picks stage 4, which holds 2.1 of the 2.3 hours. Check 2 reads its 205 GB shuffle read against the 380 GB that stage 3 scanned. A window shuffles every row it keeps, so a ratio of 0.54 fits the operation and the volume is honest. Checks 3 to 5 open stage 4's own page:
| Check | Reading on stage 4 | Verdict |
|---|---|---|
| 3. Skew | Max task 2.0 h, median 3.5 min: ratio 34 | Fails |
| 4. GC time | Task 812 spent 38 min of its 2.0 h in GC (32%); executor 41, which ran it, is the only red row | Fails, on 1 executor |
| 5. Spill | 29 GB spilled from memory as 6.4 GB on disk, all by task 812 | Fails, in 1 task |
A stage holds 80% of a job's runtime. Its code explodes an array column before a groupBy, and its shuffle write is 3.2 times its input. Its max task runs 6 times the median, and 2 executors show red GC time. Where does the diagnosis start?
- The GC time, because red executors mean the heap is too small for this stage's tasks.
- The skew, because a max-over-median ratio of 6 means one key holds far more rows than the rest.
- The shuffle volume, because a write 3.2 times the input means rows multiplied before the shuffle.
- The spill, because a stage holding 80% of the runtime almost certainly spills on every task.
About this lesson
This section is part of the The Spark UI: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.