This Job Is Slow
Concepts covered: Pipeline Monitoring, Partitioning Strategies, Small File Problem
'This job used to finish in 90 minutes and now takes 6 hours. Where do you look?' The interviewer is grading your method: which screen you open first, which number you read, and what each number rules in or out. Candidates who answer 'add more executors' before reading a metric lose the question. Skew and driver-bound stages are 2 of the common causes, and neither finishes faster with twice the executors. The Spark UI holds the handful of numbers that separate those causes, and this section reads them in order on the nightly job from the opening.
Start at the stage list
The Jobs tab shows 1 row per action with its duration, and the slow one is rarely a mystery. Click into it and the Stages tab breaks that job into stages. The columns worth reading are duration and number of tasks, then the byte columns for input and for shuffle read and write, with the spill column beside them. For the job that joins 2 TB of playback events to a 30 GB catalog and aggregates per title, last night's run looks like this.
| Stage | Work | Tasks | Duration | Shuffle write | Shuffle read | Spill (disk) |
|---|---|---|---|---|---|---|
| 1 | scan + filter events | 16,384 | 11 min | 210 GB | 0 | 0 |
| 2 | scan catalog | 240 | 1 min | 30 GB | 0 | 0 |
| 3 | sort-merge join + aggregate | 200 | 5 h 45 min | 4 GB | 240 GB | 180 GB |
| 4 | write result | 200 | 3 min | 0 | 4 GB | 0 |
Stage 3 holds 5 hours 45 minutes of the 6 hours, so every other stage is noise for now. That row already narrows the cause with 2 numbers. First, 240 GB of shuffle read landed on 200 tasks, 1.2 GB per task on average, and the stage spilled 180 GB to disk: the tasks lacked the memory to hold their share, so they sorted in chunks and merged from disk. Second, 200 is the default shuffle partition count, which nobody chose for a 240 GB shuffle. Those 2 facts are the same fact: too few partitions made each one too large for a task's memory. Whether that explains all 5 hours needs the task-level view.
Then the task summary for the slow stage
| Metric | Min | Median | 75th pct | Max |
|---|---|---|---|---|
| Duration | 8 s | 25 s | 41 s | 5 h 33 min |
| Shuffle read size | 0.3 GB | 0.6 GB | 0.9 GB | 96 GB |
| Spill (disk) | 0 | 0.2 GB | 0.5 GB | 90 GB |
| GC time | 0.2 s | 1 s | 2 s | 22 min |
Spark draws this summary for every stage, and the ratio of max to median is the most informative number in the UI. Here the median task finished in 25 seconds and the slowest took 5 hours 33 minutes, a ratio of around 800. One task read 96 GB where the median read 0.6 GB, so 1 shuffle partition held 40% of the join's bytes. That's skew: a single key or a handful of keys hash into the same partition. Skew has its own section later. The summary also shows a second problem. Even without the skewed task, the median task spilled 0.2 GB and the 75th percentile spilled 0.5 GB, so the stage would still spill after the skew is fixed. The distribution points at skew and the median points at sizing, and this stage has both.
Symptom to cause
| What you read | What it means | First change |
|---|---|---|
| Max task duration 10x the median or more | 1 partition holds far more rows than the rest: skew | salt the key or enable adaptive skew join |
| Spill on the median task | each task's share exceeds its execution memory | raise `spark.sql.shuffle.partitions` or memory per core |
| Thousands of tasks under 1 second | partitions too small, scheduling dominates | coalesce or lower the partition count |
| Scheduler delay above task duration | the driver cannot hand out tasks fast enough | fewer, larger tasks; check driver CPU |
| GC time above 10% of task time | heap pressure from many objects or wide rows | more memory per core, fewer cores per executor |
| Stage 1 shuffle write near the input size | filter ran after the shuffle, or columns were never pruned | move the filter before the join, select only needed columns |
| Same stage repeats in several jobs | the DataFrame is recomputed per action | cache it once, or write it out |
Check 2 more rows of that table on this job. Stage 1 wrote 210 GB of shuffle from a 2 TB scan, about 10%, which says the premium filter ran before the shuffle and column pruning worked; had it read 1.9 TB, the SQL tab's plan would show the filter above the join rather than inside the scan, and moving it cuts the shuffle 10 times. And the Jobs list showed 1 job, so nothing was recomputed; a script that calls count() for a log line and then write() runs the whole lineage twice and appears as 2 jobs with identical stage 1 rows.
About this lesson
This section is part of the The Spark Deep Dive: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.