Name the Usual Suspects
Concepts covered: Spark UI Diagnosis
The moment after you find the stage, the interviewer asks what could make it slow. Two answers lose. Silence loses because it shows no map of the causes. A single cause loses because, if it is wrong, the conversation has nowhere to go. A short list of named causes, each paired with the symptom that would confirm it, wins because it hands the interviewer a menu: they pick one, you go deeper, and if the first is wrong the second is already on the table. The list is short. Seven entries cover nearly every slow stage in a batch job, and the order below is the order in which each is cheapest to check.
The usual suspects, cheapest check first
| Suspect | What the stage shows | First fix |
|---|---|---|
| Input grew | Scan stage input bytes up 3x on last week; all tasks evenly slower | Confirm the growth with the data owner; add a date filter or scale out rather than tune |
| Hot key | One task's duration and shuffle read far above the median; the rest finish fast | Find the hot value; split it out, filter it, or salt it |
| Too few shuffle partitions | Every task reads over 500 MB; spill on most of them | Raise `spark.sql.shuffle.partitions` so each task reads 100 to 200 MB |
| Too many tiny tasks | Thousands of tasks of 1 to 2 seconds; scheduling overhead dominates | Coalesce, or read fewer and larger files |
| Shuffle volume | Shuffle write in tens of GB for a result of a few MB | Filter and project before the join; broadcast a small side |
| Memory pressure | Spill on most tasks with even durations; GC time over 10% of task time | More partitions first, then fewer cores per executor, then more memory |
| Slow per-row code | Even slow tasks, little shuffle, no spill; a user function in the plan | Replace the function with built-in column operations |
Input growth comes first because it costs 1 comparison: this week's bytes out of the scan stage against last week's. A job whose input tripled and whose time tripled has no bug, and tuning it wastes an afternoon that a bigger cluster or a daily partition filter would save. The hot key comes second because 1 look at the task summary settles it: the max task next to the median. The remaining 5 each need a second number before you can tell them apart, so they come after.
Apply the list to stage 2 from the narration. Input growth: stage 0's shuffle write, the events it hands to the join, is 38 GB this week against 35 GB last week, a rise of under 10%, while the time grew sevenfold, which rules growth out as the main cause. Hot key: the max task read 12.4 GB against a median of 135 MB, which is the signature. Two checks, 2 numbers, and the list has done its work, with the other 5 suspects named and set aside in the interviewer's hearing.
- 200
- 10 MB
- 128 MB
Those 3 defaults sit behind 3 of the suspects. A 40 GB shuffle over the default partition count gives each task about 200 MB, which fits a task's memory share. A 400 GB shuffle over the same count gives each task 2 GB, which spills. A catalog of 1.2 GB sits far above the broadcast threshold, so it is shuffled on the key rather than copied to every executor, and the join runs as a sort-merge join with both sides moved. A scan of 38 GB splits into a few hundred tasks as long as the files are splittable, which is why the scan stage rarely owns the clock.
Why saying them fast beats one good guess
The interviewer has a cause in mind, planted in the story. In the intro job it is a hot key that appeared in the data. If you name the hot key alone and are right, you pass the moment. If the planted cause was memory pressure, you have nothing left. If you name 4 causes with their symptoms in 30 seconds, you pass in both worlds, because the planted cause is on your list and you have told the interviewer how you would tell it apart from the others. Breadth here signals that you have debugged more than 1 slow job, and the symptom pairing signals that you would not try fixes at random on a production cluster.
- Do
- Name 3 to 5 causes, each with the UI symptom that confirms it
- Check input growth before any tuning
- Order the list by how cheap each check is
- Say which number on the stage page separates the top 2 causes
- Avoid
- Stop at 1 cause without a way to confirm it
- List causes without the symptom that distinguishes them
- Name causes the stage metrics have already ruled out
- Spend the first minute on cluster size
About this lesson
This section is part of the Make It Faster lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.