The GC Tell
Garbage collection is the JVM pausing an executor's task threads to reclaim heap from objects nothing references anymore. Every task records how long it was paused, and the UI sums those pauses in 2 places: the GC Time row of a stage's Summary Metrics, and the Executors tab's Task Time (GC Time) column, which shows an executor's total task time with its GC time in brackets.
The tell is GC time over task time. The Executors tab draws an executor's cell in red once GC passes 10% of its task time, and that line makes a good default: under 10% the heap is keeping up, and above it the executor is spending real time on memory housekeeping instead of your rows.
| Executor | Cores | Task Time (GC Time) | GC share | Storage Memory |
|---|---|---|---|---|
| 1 | 5 | 3.2 h (1.0 h) | 31% | 9.6 GB / 10.1 GB |
| 2 | 5 | 3.4 h (58 min) | 28% | 9.9 GB / 10.1 GB |
| 3 | 5 | 3.1 h (55 min) | 30% | 9.7 GB / 10.1 GB |
That is a features job where every executor is red, and the last column explains it: cached data fills nearly all the storage memory each executor has. A cached DataFrame occupies the heap for as long as it stays cached, so the collector runs against a heap mostly taken by data it can never free, and each collection reclaims less space for the tasks. You can name the cause, memory pressure from caching, before changing any setting.
3 GC patterns and what causes each
| Pattern on the Executors tab | Likely cause | First lever |
|---|---|---|
| One executor red, the rest clean | It ran the skewed task | Go back to the skew tell |
| All red, Storage Memory near full | Cached data crowding the heap | Call `unpersist()` on caches no longer read |
| All red, Storage Memory low, spill present | Each task holds too much data | More, smaller partitions |
When a single executor is red, open the slow stage's task table and sort by GC Time. The task at the top carries an Executor ID and a records count, which ties the red executor to one task and tells you whether that task was oversized.
Levers before the heap knob
Raising spark.executor.memory is the last lever, and the order of the others follows from where the pressure comes from:
- Shrink the task: More shuffle partitions, or smaller input splits, mean each task builds smaller hash tables and sort buffers, so fewer objects churn through the heap.
- Free the heap: A cache the job no longer reads keeps its memory until you call unpersist(); releasing it hands that space back to running tasks.
- Give each task more of the executor: Dropping from 5 cores to 4 on the same heap raises each task's share by a quarter, at the cost of a task slot per executor.
- Raise the heap: Every executor pays for a larger heap for the whole run, and the oversized tasks or idle cache that caused the pressure grow into it.
In the features job, the cache was a lookup table built in step 2 and never read after step 3. Releasing it after step 3 brought every executor under 6% GC and cut 40 minutes from the run without touching a memory setting.
Where the ratio misleads
Short tasks inflate the share. A task that runs 300 ms and pauses 120 ms reads as 40% GC, yet a stage of such tasks loses a few seconds in total. Judge GC on executor totals or on a stage's summed time, and treat a high GC share in sub-second tasks as noise. The opposite case deserves attention: 9% GC on a 3-hour stage is 16 minutes, worth a look even though nothing turns red.
An executor's Task Time (GC Time) reads 2.5 h (50 min). Storage Memory shows 0.4 GB of 10 GB used, and the stage it ran shows spill on every task. What is the first change?
- Raise spark.executor.memory, because 33% GC means the heap is too small for the work.
- Raise the shuffle partition count, because the cache is near empty and every task spills.
- Call unpersist() on the cached tables, because 33% GC on an executor usually comes from caching.
- Leave it, because GC share only matters once the executor's tasks run longer than a few seconds.
About this lesson
This section is part of the The Spark UI: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.