The Small-Files Problem

The in-memory sizing problem has an on-disk twin, and it is one of the most common diseases in real data platforms: the small-files problem. Take the same 100GB and store it two ways: as 800 Parquet files of 128MB, or as 100,000 Parquet files of 1MB. Identical bytes, identical schema, identical logic reading them. The second layout can make the read an order of magnitude slower, because the cost of reading data at scale is not just the bytes; it is the per-file ceremony multiplied by the file count.

Walk through what each file costs. Before reading anything, Spark has to list the directory, and listing 100,000 objects on a cloud object store is thousands of paged API calls before a single row moves. Then every file must be opened, and on an object store each open is an HTTP round trip. Parquet keeps its schema and statistics in a footer at the end of the file, so each file also costs a footer read. At 1MB per file, the metadata ceremony rivals the data itself; the reader spends its time on openings, not rows. The same ceremony over 800 files disappears into the noise.

• 100,000 files vs 800

Where do small files come from? Mostly from Spark itself, which is the uncomfortable part. Every write task emits its own file, so a job whose final stage has 5,000 near-empty partitions writes 5,000 near-empty files, and yesterday's fragmentation becomes today's layout. Incremental pipelines compound it: a streaming or hourly job appending a handful of megabytes every few minutes lays down hundreds of tiny files a day, forever. Nobody decides to create a small-files problem. It accretes, one well-intentioned append at a time, until the directory holds half a million files and every consumer of the table pays the tax.

Spark does defend itself at read time: it packs multiple small files into a single read partition rather than dutifully creating one partition per file, and it pads each extra file with a nominal open cost, 4MB by default, so that 1000 tiny files do not masquerade as cheap input. Packing keeps the task count sane, and it is why the symptom is not always 100,000 tasks. But packing cannot refund the listing calls, the open round trips, or the footer reads; those are paid per file no matter how files are grouped into partitions. The only real fix is to change the layout on disk: rewrite many small files into fewer right-sized ones, which is exactly the compaction work the advanced tier covers.

The ticket this arrives as reads something like: why does reading 10GB take 40 minutes? The bytes are small; the file count is the problem, and it is invisible in the code. When a read is mysteriously slow, make checking the file count and average file size a reflex, before touching a single transformation. A healthy table holds files near the same 128MB target as partitions, and for the same two reasons: enough substance to amortize per-file ceremony, small enough to split work evenly.

About this lesson

This section is part of the Partitioning: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.