Compacted Topics
Some topics carry state rather than events. A topic of customer addresses keys each record by customer id and carries the current address as the value. The 5th address c-7 sent makes the earlier 4 worthless, but the delete policy would also drop the 5th once it turned 7 days old, and a reader building a table of addresses would then find c-7 missing. cleanup.policy=compact changes what a partition keeps: for each key, the record with the highest offset is never removed, and older records with the same key may be. A reader starting at offset 0 receives every key's latest value, however old it is, and the partition's size tracks the number of distinct keys rather than the number of writes.
One partition, before and after a cleaning pass
The pass left 4 records, 1 per key, and each kept the offset it was written at. Offsets 0 to 3 no longer exist, and a reader that asks for offset 1 is handed the record at 4, the next offset the partition has. The survivors stay in their original order, so c-2 at 4 still comes before c-7 at 5. Compaction never creates or moves a record; it only leaves some out when it rewrites a segment.
Where the cleaning stops
The work is done by the log cleaner, a background thread on the broker, and it rewrites closed segments only: it reads a closed segment, copies the records that are the latest for their key into a new segment file, and swaps the new file in for the old. The active segment is never cleaned, and a closed segment waits its turn, so at any moment a partition is a cleaned part at the front and an uncleaned tail at the back where a key can appear several times. A later lesson covers how the cleaner decides which segments to visit and when. Whatever that schedule is, the tail is always there, so a reader must expect repeated keys and keep the last value it sees for each.
| part of the partition | segments | what a key looks like there |
|---|---|---|
| cleaned head | closed segments the cleaner has rewritten | at most 1 record per key |
| uncleaned tail | closed segments not yet visited, plus the active segment | every write, repeats and all |
After a pass, a segment that held 1 GiB of writes for 10,000 keys may hold 10 MiB, so the cleaner also merges small cleaned segments into one file to keep the directory from filling with tiny segments. A compacted partition's disk use therefore settles at roughly the size of 1 value per key, plus whatever the tail holds at the time.
Cleaning is per partition, so the promise of 1 latest record per key holds only when every record for a key lands in the same partition, which the key itself arranges and the last section returns to.
What a compacted topic demands
The policy is set per topic, at creation with kafka-topics.sh --create --bootstrap-server localhost:9092 --topic customer-addresses --partitions 6 --config cleanup.policy=compact or later with kafka-configs.sh. A topic may carry both policies, cleanup.policy=compact,delete: the cleaner keeps the latest value per key, and the retention thread still removes segments whose newest record is older than retention.ms, so a key that stops being written vanishes after the window.
- cleanup.policy=delete
- Every record is an event and every one matters
- Size is bounded by time or by bytes
- A reader starting at 0 sees the last window
- Keys are optional
- cleanup.policy=compact
- Each record is the state of one key
- Size is bounded by the number of keys
- A reader starting at 0 sees every key's latest value
- Keys are required
The reader-side rule that falls out of all this is short: apply records in offset order and let the later record for a key overwrite the earlier one. A reader written that way is correct against the cleaned head, the uncleaned tail, and a partition the cleaner has never visited, because in every case the record with the highest offset for a key is the one that wins.
About this lesson
This section is part of the The Log: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.