Too Few Large Partitions
The opposite failure is a dataset cut into too few, too-large pieces: 100GB in 8 partitions of 12GB each. The first casualty is parallelism. Spark runs one task per partition, so this job can only ever use 8 cores at once. On a 64-core cluster, 56 cores sit idle for the entire run, and here is the trap that catches teams over and over: adding more hardware does nothing. Doubling the executors on an 8-partition job buys you more idle cores. The ceiling is the partition count, and no amount of spend raises it.
The second casualty is memory. A task has to hold its partition's working data while it processes it, and a task's slice of executor memory is nowhere near 12GB on a typical setup. When the partition does not fit, one of two things happens. If the operation can spill, the task writes overflow to disk and reads it back, turning memory work into slow disk work. If it cannot, or the pressure is severe enough, the executor dies with an out-of-memory error. Then it gets worse: Spark retries the failed task, and the retry processes the same oversized partition with the same memory, so it usually dies the same way. Giant partitions do not just risk failure; they make failure expensive and repetitive.
Large partitions also create stragglers. If 7 partitions hold 4GB and one holds 40GB, 7 tasks finish and the whole job then waits on the one task chewing through the boulder, because a stage is only done when its slowest task is. One oversized partition sets the runtime for the entire stage, while the rest of the cluster watches. You will meet this shape again in the data-skew lesson, where the imbalance comes from the data's key distribution; here the point is simpler: even with perfectly uniform data, choosing too few partitions concentrates work into indivisible lumps that nothing downstream can parallelize.
- Too many small partitions
- 100,000 tiny tasks, milliseconds each
- Overhead dominates useful work
- Driver strained by task bookkeeping
- Slow, but quietly: nothing crashes
- Too few large partitions
- Only a handful of tasks can run
- Cores idle no matter the cluster size
- Spill, OOM kills, expensive retries
- Fails loudly, or crawls on one core
Notice that the two failure modes present completely differently. Fragmentation is quiet: everything succeeds, slowly. Under-partitioning is loud: spills, dead executors, retry storms, a stage stuck at 7 of 8 tasks for an hour. But they are the same disease, a partition count that stopped matching the data volume, and the cure is the same arithmetic from the first section: bytes divided by 128MB, sanity-checked against the cluster's cores. Somewhere between 8 boulders and 100,000 crumbs is a count where tasks are meaty enough to amortize their overhead and small enough to fit in memory.
A 90GB dataset is processed as 6 partitions on a 48-core cluster. The stage shows 6 long-running tasks and repeated executor OOM kills. What is the root problem?
- The cluster is too small; adding executors would fix it.
- The data is skewed toward a few hot keys.
- The partition count is far too low: 90GB over 6 partitions is 15GB per task, which idles 42 cores and overflows task memory.
Carry the pair of symptoms with you as one diagnostic: a stage that is slow with a huge task count and tiny task times is over-partitioned; a stage that is slow with a tiny task count, idle cores, and spill or OOM is under-partitioned. Between them, those two checks explain a remarkable share of every slow Spark job you will ever be handed. As a working floor, keep the partition count at least a few multiples of the cluster's core count, so the work runs in several even waves and no single task can hold the whole stage hostage.
About this lesson
This section is part of the Partitioning: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.