Compaction in Table Formats
A table format replaces 'the files in this directory' with a log of snapshots. Each snapshot points to manifests, and each manifest lists data files, recording for each one its partition values and row count along with column statistics. A reader resolves one snapshot and reads exactly the files it lists. Compaction becomes a commit like any other: it adds the new large files and removes the old small ones in one swap of the current-snapshot pointer.
- snapshots: snapshot_id bigint PK, parent_id bigint, operation varchar, committed_at timestamp
- manifests: manifest_path varchar PK, added_snapshot_id bigint FK, added_files integer, removed_files integer
- data_files: file_path varchar PK, manifest_path varchar FK, partition_hour timestamp, record_count bigint, file_size_bytes bigint, column_stats map
- manifests.added_snapshot_id references snapshots.snapshot_id (N:1)
- data_files.manifest_path references manifests.manifest_path (N:1)
Simplified metadata: real formats also keep a manifest list per snapshot, and a manifest written by one snapshot stays shared by the snapshots after it.
Directory swap versus commit
- Compaction on a plain directory
- Writes 5 new files, then deletes 12,000 old ones
- A reader listing in between sees both and counts every trip twice
- A retry after a partial delete can't tell which files it already removed
- Every reader still lists keys to plan
- Compaction as a table-format commit
- New files stay invisible until the snapshot pointer swaps
- Readers see the old 12,000 or the new 5, never both
- A failed or retried job leaves only unreferenced files for cleanup
- Planning reads file entries from manifests, with no listing
Schema change separates them too. A directory compaction that infers its schema from the first file it opens drops a column added at noon if that file came from the morning. A table format that tracks columns by stable ID reads every file through the table's current schema, so morning files surface the new column as null, renamed columns keep their data, and the rewritten files carry the current schema. Formats that resolve columns by name need column mapping enabled before a rename is safe.
Optimistic concurrency and the compaction loop
Compaction starts from snapshot S, spends 4 minutes rewriting an hour, and meanwhile 4 minute appends commit. Before its commit lands, the format validates that every file the job removes is still live and unmodified in the latest snapshot. What counts as a conflict decides whether compaction ever finishes on a busy table.
| Concurrent operation | Conflicts with the rewrite? | Reason |
|---|---|---|
| Append to the same hour | No | New files are added; none of the removed files changed |
| Row-level delete touching a rewritten file | Yes | The new large file would resurrect the deleted rows |
| Second compaction of the same files | Yes | Both commits remove the same files; the later one must retry |
| Column added to the schema | Depends on the format | No conflict where columns have stable IDs and only data files are validated; formats that validate the whole table metadata reject the commit, and the rewrite retries from the new snapshot |
Row-level deletes cause the worst conflicts. A privacy job that deletes rows across all hours every 15 minutes conflicts with any 4-minute rewrite that overlaps its targets, and a compaction that retries from scratch can lose that race on every attempt. Commit in file groups, for example 10 per run, so a conflict discards one group's work, and schedule row-level deletes and compaction into separate windows per partition.
Metadata needs compaction too
Every minute append is a commit, and each commit writes at least 1 manifest, so trip_events gains 1,440 snapshots and 1,440 manifests a day. Planning a scan of yesterday opens all 1,440 before touching data. Manifest rewriting merges them into a handful grouped by partition, the same bin-packing applied one level up.
The update strategy sets how much more work exists. Copy-on-write rewrites each touched data file at update time: more write amplification per update and no extra small files. Merge-on-read writes small delete files at update time and defers the rewrite, so every read must apply those delete files until compaction rewrites the data files and drops them from the next snapshot. A merge-on-read table therefore needs a compaction cadence matched to its update rate.
Storage drops only after snapshots expire
Compaction doesn't shrink the bill on the day it runs. The 12,000 small files of each hour stay referenced by older snapshots, which time travel needs, until those snapshots expire. With 7 days of retention, the table keeps an extra 210 GB of superseded files: 7 days of 30 GB. Snapshot expiry deletes files no live snapshot references. Orphan cleanup handles the other leftover, files from failed writers that no snapshot ever referenced, and it must only remove files older than the longest possible write, such as 3 days, or it deletes the output of a job still running.
About this lesson
This section is part of the Small Files and Compaction: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.