"Just Add More Memory"
Concepts covered: Spark UI Diagnosis
Add more memory is the most common first answer to a slow job, and the interviewer hears it as a reflex. Memory in an executor does 1 thing for a running task: it holds the task's working set, the hash table or sorted buffer that a join or an aggregate builds, so the task avoids writing that buffer to disk. Spark reserves 300 MB of the heap, gives a fraction of the rest (0.6 by default) to execution and storage together, and divides that between the tasks running on the executor. When a task's buffer outgrows its share, it spills: it writes the buffer to local disk, continues, and merges the pieces later. More memory means fewer spills. It changes nothing else about the task: the rows it must process, the bytes it must read over the network, the code it must run on each row.
What memory fixes, and what it leaves alone
| Symptom | Does more memory help? | Why |
|---|---|---|
| 1 task far slower than the median | No | The hot task still reads its 12.4 GB alone; a bigger heap shortens its spill, and its row count stays the same |
| Most tasks spill, durations even | Yes | Each task's buffer now fits, so the spill disappears and the disk write with it |
| Shuffle write far above output size | No | The bytes still cross the network; memory does not reduce them |
| Thousands of 1-second tasks | No | Scheduling overhead has nothing to do with heap size |
| Input tripled since last week | No | Three times the rows need 3 times the work; memory only moves where the work happens |
| GC time over 10% of task time | Sometimes | A larger heap can lengthen each pause; more partitions or fewer cores per executor often do more |
Run the memory answer against stage 2. The job's executors have 8 GB each with 4 cores, so each task gets roughly 1 GB of execution memory. The median task reads 135 MB and never spills, so 199 of the 200 tasks have nothing to gain. The hot task reads 12.4 GB and spilled 24 GB over its 2 h 48 min. Doubling the executor to 16 GB gives that task roughly 2 GB, still a sixth of what it reads, so it still spills, somewhat less, and still runs for well over 2 hours. The stage that owned the clock still owns it, and the cluster now runs fewer executors per node for the same bill.
That profile is memory pressure, and it is a real cause: a stage where every task reads 600 MB on a 1 GB share will spill on all 200 tasks, and the stage runs perhaps twice as long as it should. Even then, memory is the second fix to name. Raising the shuffle partition count from 200 to 800 drops each task to 150 MB on the same hardware, with no new bill, and the only cost is 600 more small tasks to schedule. The interviewer wants to hear that you know the free fix before the paid one.
spark.conf.set("spark.sql.shuffle.partitions", 800)The honest version of the memory answer
There is a case where memory is the first fix, and it is worth separating from the slow case: the job that dies. An executor lost with a container-killed message, or an out-of-memory error in the driver after a collect, is a size problem rather than a speed problem, and the stage metrics will show tasks that never finished rather than tasks that ran long. In the interview, ask which one the job did. A job that finishes late has a profile to read. A job that fails has a limit to raise, and even then the question of why one task needed that much memory, which is usually the same hot key, is worth a sentence before the limit goes up.
- Saying 'add memory' first
- Signals a reflex rather than a reading of the stage
- Costs money on every run
- Leaves a hot key, shuffle volume and tiny tasks untouched
- Gives the interviewer nothing to probe
- Saying it after the stage profile
- Signals that the symptom pointed there
- Comes after the free partition fix
- Names the task-level number it will change: spill
- Ends with the re-run that shows spill at 0
The cost side deserves 1 sentence in the room. A node with 64 GB fits about 7 executors at 8 GB each, once the operating system and per-executor overhead take their share, and about 3 at 16 GB. Doubling memory on a fixed cluster therefore cuts the executor count by more than half, and a stage with 200 evenly sized tasks then runs on fewer cores and gets slower. The memory answer can lose time as well as money, which is the part the reflex never mentions.
About this lesson
This section is part of the Make It Faster lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.