Too Many Small Partitions
The first failure mode is a dataset cut into far too many small pieces: 100,000 partitions of a few hundred kilobytes each. Nothing crashes. The job just runs mysteriously slowly, because every one of those 100,000 tasks pays the same fixed ceremony regardless of how little data it holds. The scheduler has to pick an executor for it, serialize the task, send it over the network, launch it, and collect its status when it finishes. Each of those steps costs a few milliseconds. When the useful work inside the task is also a few milliseconds, the ceremony is no longer a rounding error; it is the job.
Put numbers on it. Suppose per-task overhead totals around 100 milliseconds and each tiny partition needs 20 milliseconds of actual computation. Five sixths of all cluster activity is now administration. A job that should take a minute takes six, and profiling the transformations finds nothing, because the transformations were never the problem. The driver suffers too: it is a single process tracking the state of every task, and a stage with hundreds of thousands of tasks means hundreds of thousands of launch decisions and completion events flowing through one JVM. Enough tiny tasks can make the driver itself the bottleneck while every executor sits mostly idle.
| What each task pays | Typical cost | Why 100,000 partitions hurts |
|---|---|---|
| Schedule and serialize the task | A few milliseconds | Paid once per task, so 100,000 times |
| Launch and report completion | A few milliseconds | Every event flows through the one driver |
| Useful work in a tiny partition | Often under 20 milliseconds | Dwarfed by the fixed costs around it |
Where do too-many-small partitions come from? 3 places, mostly. Reading a directory full of small files starts the job fragmented, a problem big enough to get its own section shortly. A filter that discards most of the data shrinks the bytes but not the partition count, so 1000 healthy partitions become 1000 near-empty ones. And a shuffle produces a fixed number of output partitions regardless of data volume, so shuffling a small dataset across a large partition count manufactures fragments. In each case the count stopped matching the bytes, and the fix is the same idea: bring the count back in line with the data.
orders.filter(F.col("status") == "refunded").collect()| order_id | status | amount |
|---|---|---|
| 1 | shipped | 40 |
| 2 | shipped | 25 |
| 3 | refunded | 60 |
| 4 | shipped | 35 |
| 5 | shipped | 50 |
| 6 | shipped | 20 |
| 7 | shipped | 45 |
| 8 | shipped | 30 |
| 9 | refunded | 15 |
| 10 | shipped | 55 |
| 11 | shipped | 70 |
| 12 | shipped | 65 |
The tell in the Spark UI is distinctive once you know it. A stage with an enormous task count where every task finishes in milliseconds, and where the sum of task time is small compared to how long the stage took on the wall clock. The gap between those two numbers is scheduling overhead, and it is the signature of fragmentation. You will learn the repair tools, coalesce and repartition, in the intermediate tier; the beginner skill is noticing the shape at all, because it hides well. Nothing fails, nothing spills, and the logs look clean. The job is just several times slower than the data justifies, and the meter is running.
That meter is the real cost. Clusters are billed by the hour whether cores do useful work or ceremony, so a job that spends most of its time on task administration is paying full price for a mostly idle cluster, every night, in the dark. Fragmentation is rarely dramatic enough to page anyone, which is exactly why it survives in production pipelines for years. When you audit a pipeline's spend, start with its most task-heavy stages: the fragmented ones are usually paying the largest share of the bill for the smallest share of the work.
About this lesson
This section is part of the Partitioning: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.