The Skew Tell
The stage page's Summary Metrics table gives each metric's spread across the stage's tasks: Min, 25th percentile, Median, 75th percentile and Max. The skew tell is a single division, the Max of the Duration row over its Median. The median is the right baseline because one straggler drags a mean upward while barely moving the middle task.
| Metric | Min | 25th | Median | 75th | Max |
|---|---|---|---|---|---|
| Duration | 48 s | 2.9 min | 3.5 min | 4.1 min | 2.0 h |
| GC Time | 1 s | 4 s | 5 s | 7 s | 38 min |
| Shuffle Read Size / Records | 160 MB / 1.6M | 182 MB / 1.8M | 190 MB / 1.9M | 199 MB / 2.0M | 6.6 GB / 66M |
| Spill (Disk) | 0 B | 0 B | 0 B | 0 B | 6.4 GB |
For stage 4 of the ranking job, 2.0 hours over 3.5 minutes gives 34. Duration alone cannot say why the task was slow, so divide a second row the same way: 66 million shuffle read records over 1.9 million gives 35. The 2 ratios match. The slowest task took 34 times as long because it read 35 times the rows, nearly all of them for a single title_id. That is data skew.
| Max over median | What it means |
|---|---|
| Under 2 | Ordinary variation between tasks |
| 2 to 5 | Open the task table and read the slowest 10 tasks |
| Above 5 | Skew or a sick node: decide which before changing code |
These bands are a working heuristic from practice; Spark itself flags none of them.
Data skew or a slow node
A high duration ratio has 2 causes that need opposite responses. The records ratio separates them, and the task table's Executor ID and Host columns confirm the call.
- Data skew
- Records ratio close to the duration ratio
- The slow tasks read far more shuffle records
- Several slow tasks sit on different executors
- A rerun is slow at the same task index, wherever it lands
- Slow node
- Records ratio near 1
- The slow tasks read an ordinary share of records
- The slow tasks share one Executor ID and Host
- A rerun whose tasks land elsewhere runs at normal speed
Here is the task table from a different job's slow stage, sorted by Duration, longest first. Its median task runs 11 s, so the duration ratio is 13, which looks like skew until you read the last column. The first 40 rows all carry executor 17; these are the top 3 and row 41:
| Row | Task | Executor ID | Host | Duration | Shuffle Read Records |
|---|---|---|---|---|---|
| 1 | 1311 | 17 | 10.0.3.21 | 2.4 min | 1.2M |
| 2 | 1290 | 17 | 10.0.3.21 | 2.3 min | 1.1M |
| 3 | 1347 | 17 | 10.0.3.21 | 2.3 min | 1.1M |
| 41 | 1318 | 6 | 10.0.1.9 | 12 s | 1.1M |
Records sit near 1.1 million for every task, and the slow ones share executor 17. The Aggregated Metrics by Executor table on the same page shows it from the other side: executor 17 spent 1.5 hours of task time on 40 tasks, while every other executor spent under 9 minutes on its 40. The data is even and one machine is slow, from a failing disk, a noisy neighbour or a throttled CPU. Salting would change nothing; the fix is to report the node and get it out of the pool.
A stage's Summary Metrics show a median Duration of 20 s and a max of 4.0 min. Shuffle Read Records show a median of 800K and a max of 840K. The 6 slowest tasks all ran on executor 9. What do you conclude?
- Data skew: a duration ratio of 12 sits well above 5, so the stage needs a salted key.
- Data skew: the 6 slow tasks read the largest shares, so the hottest keys landed on executor 9.
- No problem: a max of 4 minutes is short enough that the spread between tasks does not matter.
- A slow node: records barely vary and the slow tasks share executor 9, so the machine is the cause.
The animation shows the data-skew side on a sort-merge join. Title 7 appears in 10 of the 15 plays, so whichever reduce task hash partitioning assigns to key 7 receives at least 11 rows, the 10 plays plus the title itself, while the other tasks split the remaining handful.
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
plays.join(titles, "title_id").count()| title_id | profile_id |
|---|---|
| 7 | 101 |
| 7 | 102 |
| 3 | 103 |
| 7 | 104 |
| 7 | 105 |
| 5 | 106 |
| 7 | 107 |
| 7 | 108 |
| 9 | 109 |
| 7 | 110 |
| 3 | 111 |
| 7 | 112 |
| 7 | 113 |
| 5 | 114 |
| 9 | 115 |
| title_id | title |
|---|---|
| 3 | Cooking Duel |
| 5 | Deep Sea |
| 7 | Big Hit |
| 9 | Pilot Night |
When the tell reads data skew, the levers are ones you already have: salt the hot key, isolate it and process it separately, or filter it out first if the hot key is NULL. For a sort-merge join, adaptive execution splits any partition larger than both 5 times the median and 256 MB on its own. A window like stage 4's gets no such help, which is why its 66 million rows still ran on a single core.
Try the tell on the practice data. The cell counts events per user, then counts how many users sit at each event total. Read 2 values: the events figure where the running total of users first passes half of all users, which is the median, and the events figure in the last row, which is the max. Their ratio is the skew tell for any shuffle keyed on user_id.
About this lesson
This section is part of the The Spark UI: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.