Why Small Files Kill Performance

Here is the estimate worked through. Yesterday's 288,000 files at 2 requests each make 576,000 requests, and at 20 ms apiece that is 11,520 seconds of waiting. Spread over 100 slots, it comes to 115 seconds before a single row is decoded, on a scan whose data takes roughly 3 seconds to move.

Every file carries a fixed price

Cost	Paid once per	How it grows with file count
Listing	page of up to 1,000 keys	Serial list calls during planning on a plain directory
Footer read	file	1 request, then decoding schema and column statistics
Data read	file	At least 1 more request and its first-byte latency
Task scheduling	split	Engines pack small files into splits, so task count stays bounded
Metadata entry	file	A manifest or catalog row the planner must read and filter
Encoding	file	Dictionaries and statistics restart, so compression weakens

None of these depend on how many bytes the file holds. A 110 KB file and a 256 MB file each cost 1 footer read, at least 1 data request and 1 metadata entry, so a scan costs roughly its bytes divided by bandwidth plus its files multiplied by a fixed price.

One day of trip_events in 3 layouts

Fix the assumptions and say them aloud: 20 ms first-byte latency per request, 2 requests per file, 100 reader slots each pulling 100 MB/s, and 50 ms per list call returning 1,000 keys. Moving 30 GB through those slots takes 3.1 seconds in every layout. What changes is everything paid per object.

Layout	Files per day	Average file	Requests	Request wait	Listing	Total
1-minute batches, 200 tasks	288,000	110 KB	576,000	115.2 s	14.4 s	133 s
1-hour batches, 200 tasks	4,800	6.4 MB	9,600	1.9 s	0.25 s	5.2 s
Compacted to 256 MB	120	256 MB	240	0.05 s	0.05 s	3.2 s

The slow layout does no extra useful work: it reads the same rows and the same bytes. It spends 87% of its time waiting on request latency and 11% listing. Say that sentence in the interview, because it tells the interviewer which term a fix must attack, and it shows why a bigger cluster barely helps: doubling slots halves the wait to 58 seconds, while listing, which runs serially on the coordinator, stays at 14.4.

Requests also cost money. At an assumed $0.0004 per 1,000 GET requests, one scan of yesterday costs $0.23 in the 1-minute layout and $0.0001 compacted. Dashboards on refresh timers plus analysts can drive 400 such scans a day.

$33,600
$14

Try it. Extend the 1-minute layout to a 30-day scan under the same assumptions. How many files and requests does it touch, and how many minutes does it spend waiting on requests?

Answer: 8.64 million files, 17.28 million requests and 3,456 seconds of wait, roughly 58 minutes, plus 8,640 list calls that take 7.2 minutes on their own.

Why split packing does not rescue you

Engines do not launch a task per file. They pack small files into splits up to a target size, say 128 MB, so the 30 GB day becomes about 240 tasks instead of 288,000. That fixes scheduling overhead and leaves request latency untouched. Each split now holds 1,200 files, and a task opens them one after another: 2,400 requests at 20 ms is 48 seconds per task, and 240 tasks of 48 seconds on 100 slots lands back at the same 115 seconds. Readers that prefetch several files at once shorten this, at the price of memory per task and more pressure on the per-prefix request limits that object stores enforce.

Costs that never show up in a latency chart

Second-order damage

Weaker compression. A 110 KB file restarts its dictionaries and stores its own schema and statistics, so the same rows take more bytes than inside a 256 MB file.
Pruning that costs more than it saves. The planner tests a filter against 288,000 sets of min/max values, and files cut from arrival order span nearly the full range of every column, so few get skipped.
Planning metadata. A table format records each file as a manifest entry: 288,000 a day, 8.64 million for a 30-day scan, all read before the first task starts.
Writer overhead. Each file is opened, flushed, closed and recorded at commit, so the writer spends 200 file lifecycles a minute on 21 MB of data.
Downstream spread. A job that reads trip_events and writes 1 output file per input split copies the layout into the next table.

About this lesson

This section is part of the Small Files and Compaction: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.