The Spill Tell
Spill is a task writing part of its sort or aggregation buffer to local disk because execution memory ran out, then reading it back to finish. The stage page gives it 2 columns, Spill (Memory) and Spill (Disk), and shows them only when some task in the stage spilled. If the columns are absent, no task spilled; if they appear at any size, that is a finding.
Spill (Memory) is how large the spilled data was in its in-memory form; Spill (Disk) is the same data serialized and compressed on disk, so the disk figure is the extra write and read the task paid for. On stage 4 of the ranking job, task 812 spilled 29 GB from memory as 6.4 GB on disk.
Why spill means the task is too big
A task's working space comes from arithmetic you have met before. Take an executor with a 16 GB heap and 5 cores. Spark reserves 300 MB, gives 60% of the rest to the unified execution and storage pool, and divides the execution side among the tasks running at once:
- 1.9 GB
A shuffle partition of 2 GB on disk expands well past that once it is deserialized into hash tables and sort buffers, so the task spills. Smaller partitions change the arithmetic, and the shuffle partition count is the cheapest dial: it adds no cost per executor and acts only on the stages that shuffle.
A worked fix
A daily aggregation reads 400 GB of shuffle data with spark.sql.shuffle.partitions at 200, so each task fetches about 2 GB. Divide the shuffle read by a target of 200 MB per partition to get the new count: 400 GB over 200 MB is 2,000.
spark.conf.set("spark.sql.shuffle.partitions", 2000)Adaptive execution coalesces post-shuffle partitions that come out smaller than its 64 MB advisory size, so overshooting the count costs little. Undershooting costs the spill you started with.
Which lever, by where the spill sits
| Where the spill sits | First lever |
|---|---|
| Every task in a shuffle stage, about evenly | Raise `spark.sql.shuffle.partitions` toward 100 to 200 MB per partition |
| A few tasks; the rest show 0 B | Skew: run the skew tell and fix the hot key |
| Tasks of a scan stage | Lower `spark.sql.files.maxPartitionBytes` from its 128 MB default |
| Every task, even at 100 MB partitions | Fewer cores per executor, then a larger heap |
The second row is why spill comes after skew in the diagnosis. Task 812's 29 GB spill sat in a single task. Raising the partition count would have left the hot title_id in one partition, because a key never splits across partitions, and the spill would have stayed where it was.
- Do
- Treat any spill figure as a finding, whatever its size
- Size shuffle partitions as shuffle read over 100 to 200 MB
- Compare the Median and Max spill to see whether spill is even
- Re-read the stage page after a change; the spill columns should vanish
- Avoid
- Raise executor memory before trying more partitions
- Add partitions when a single task holds all the spill
- Call 310 GB of disk spill small because the stage still finished
- Tune a scan stage's spill with the shuffle partition setting
A stage shows Spill (Disk) with Min 0 B, Median 0 B and Max 14 GB. Shuffle read per task has a median of 120 MB. Which lever comes first?
- Raise spark.sql.shuffle.partitions, because any spill means the partitions are too large.
- Raise spark.executor.memory, because 14 GB of spill needs more room than the tasks have now.
- Fix the skew, because the median task spills nothing and all the spill sits in the Max tail.
- Lower spark.sql.files.maxPartitionBytes, because spill in the Max column means oversized input splits.
About this lesson
This section is part of the The Spark UI: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.