Preventing Small Files at the Source
Concepts covered: Small File Problem
Compaction repairs a layout after the fact and pays a rewrite for it. Prevention changes how many files the writer creates in the first place, and it comes down to one bound.
Files written per commit are at most the writer tasks times the partitions each task touches. The baseline writer has 200 tasks, each touching 1 hour partition, committing every minute: 200 files a minute. Every fix below shrinks one factor or the number of commits.
| Change | Files per hour | Average file | What it costs |
|---|---|---|---|
| Baseline: 1-minute commits, 200 tasks, rows in arrival order | 12,000 | 110 KB | Nothing up front; every reader pays |
| Shuffle rows by partition_hour before writing | 60 | 21 MB | A shuffle per batch; 1 task carries the whole hour |
| Also stretch commits to every 5 minutes | 12 | 107 MB | Data is up to 5 minutes old instead of 1 |
| Also rewrite each closed hour at 256 MB | 5 | 256 MB | A second write of every byte; curated data lags up to 1 hour |
Read the table as a menu with prices. The first change spends no freshness and costs a shuffle, the second spends freshness, the third spends storage and compute. A strong answer picks the cheapest combination that meets the consumer's stated freshness, instead of applying all 3 by reflex.
Distribute rows before the write
Shuffling by the partition column sends each hour's rows to one task, which writes 1 file per commit. The risk is skew. At 21 MB a minute one task copes, but a holiday hour at 4 times normal volume puts a hot partition on a single writer. The writer rolls to a new file at 256 MB, so no file outgrows target, and a very large partition is split across several tasks by range on a second column, which costs a few extra files and keeps throughput.
Backfills multiply the same bound
Batch jobs hit the same bound with bigger factors. A 30-day backfill with 200 tasks reading rows in source order touches every hour from every task: up to 200 times 720, or 144,000 files of about 6.4 MB. Shuffling by hour first and rolling files at 256 MB writes 5 per hour, 3,600 in total. Engines that coalesce shuffle output toward a target partition size reach a similar result automatically, provided the target is set in bytes rather than left at a default task count.
Over-partitioning sets a floor compaction cannot lift
Suppose the team partitions by hour and by city across 600 cities. The bound now reads 200 tasks times up to 600 partitions each: 120,000 files from a single 1-minute commit. Compaction cannot repair most of it, since a rewrite never merges files across partitions and most partitions hold too few bytes to fill a target file. City volume is skewed, so a few large cities fill whole files while most of the 600 sit well below the average. The fix is a coarser key such as hour or day, with city order moved inside the files through sort compaction, where it prunes without multiplying files.
- 14,400
- 2.1 MB
Split freshness from layout with a raw tier
- trip_queue (queue, message queue)
- minute_writer (transform, streaming job); freshness: 1 min; parallelism: 200 tasks
- trip_events_raw (bronze, raw table, 3-day retention); idempotency: atomic append commit
- hourly_rewrite (transform, hourly batch job); retries: 3; backfill: rebuild hours touched since last run; idempotency: overwrite hour, dedupe on event_id
- trip_events (silver, curated table, 256 MB files); alerting: small files per hour over 50
- trips_dashboard (consumer, view over raw and curated)
- trip_queue -> minute_writer
- minute_writer -> trip_events_raw
- trip_events_raw -> hourly_rewrite
- hourly_rewrite -> trip_events
- trip_events -> trips_dashboard
- trip_events_raw -> trips_dashboard
- Sources: trip_queue
- Raw: minute_writer, trip_events_raw
- Curated: hourly_rewrite, trip_events
- Serving: trips_dashboard
The minute writer lands data in a raw table kept for 3 days, and an hourly batch job rebuilds every hour that received rows since its last run. Its input is the hour's existing curated rows plus the new raw rows, deduplicated on event_id, so an event arriving 5 days late still merges with an hour whose raw rows have expired. The rebuild overwrites the whole hour partition in one atomic commit, which makes a retry produce identical output. Consumers that need the last minutes read a view that unions the raw current hour with curated history; everyone else reads curated only.
About this lesson
This section is part of the Small Files and Compaction: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.