How Do You Size the Cluster?
Concepts covered: Batch Processing, Compression & Encoding, Partitioning Strategies
'We have 1 TB a day, 16 nodes with 16 cores and 64 GB each, and a 30 minute window. How would you configure it?' Several configurations are defensible, and the interviewer knows it; what they score is the chain of arithmetic from input bytes to executor shape, and whether each number you pick has a reason behind it. The chain runs in a fixed order: executor shape, then memory per task, then partition count, then waves against the deadline.
Executor shape
Set aside 1 core and 1 GB per node for the operating system and the cluster manager's agent. That leaves 15 cores and 63 GB. The long-standing guidance caps an executor at 5 cores, because file system client throughput per JVM flattens past around 5 concurrent readers and because a heap large enough to feed 15 tasks pauses for garbage collection long enough to show up as task time. Dividing 15 cores by 5 gives 3 executors per node; 63 GB divided by 3 gives 21 GB per executor container. Spark reserves spark.executor.memoryOverhead outside the heap, 10% of executor memory or 384 MB, whichever is larger, for off-heap buffers and the Python worker if you use one, so request a 19 GB heap and the container lands at 21 GB. Across 16 nodes that is 48 executors and 240 task slots.
| Shape per node | Executors | Cores each | Heap each | What goes wrong |
|---|---|---|---|---|
| 1 fat executor | 1 | 15 | 57 GB | long GC pauses; 1 JVM failure loses 15 slots and all its shuffle files |
| 3 balanced | 3 | 5 | 19 GB | the common choice |
| 15 thin | 15 | 1 | 3.8 GB | 15 copies of every broadcast, no slot sharing, overhead x15 |
Memory per task
A 19 GB heap gives a task far less than 19 GB. Spark holds back 300 MB, then applies spark.memory.fraction, 0.6 by default, to the rest: (19 GB - 300 MB) x 0.6 = 11.2 GB of unified memory. Execution and storage share that pool. The executor's 5 concurrent tasks share it too, so each one can count on about 2.2 GB for sorting and hashing before it spills, and less if cached partitions occupy the storage half. That's the number the slow-job section was missing: a shuffle partition near 1.2 GB, once deserialized into objects with a sort buffer on top, spills against a 2.2 GB ceiling, and a 128 MB to 256 MB partition does not.
Partitions and waves
| Step | Arithmetic | Result |
|---|---|---|
| Input tasks | 1 TB / 128 MB splits | 8,192 |
| Shuffle bytes after the 10% filter | 1 TB x 0.1 | 100 GB |
| Shuffle partitions at 200 MB each | 100 GB / 200 MB | 500, rounded to 480 |
| Task slots | 48 executors x 5 cores | 240 |
| Stage 1 waves | 8,192 / 240 | 34.1 |
| Stage 2 waves | 480 / 240 | 2 exactly |
| Stage 1 time at 20 s per task | 35 waves x 20 s | 11.7 min |
Both rounding decisions in that table have a reason. The shuffle partition count is rounded to a multiple of the slot count, because 500 partitions on 240 slots run 2 full waves and then 20 tasks on a cluster that is 92% idle; 480 fills every slot in both waves. And stage 1 is quoted as 35 waves rather than 34.1 because the fractional wave runs at the tail at the full cost of a wave. At 20 seconds a task the stage takes under 12 minutes, which fits the 30 minute window with room for the join; if the measured task time comes back at 60 seconds, the same arithmetic says 35 minutes and the answer changes to more nodes or a smaller input. Set the count with spark.sql.shuffle.partitions, or set it high and let adaptive execution coalesce small partitions toward its 64 MB advisory size, which absorbs a filter rate that changes from day to day.
About this lesson
This section is part of the The Spark Deep Dive: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.