When Compaction Runs

Concepts covered: Log compaction

The intermediate lesson ended with a promise: a compacted topic keeps the latest value for every key. It said nothing about when. Right after a producer writes a new value for key u42, the partition holds the old value and the new one, and a reader that starts at offset 0 sees both. Removing the old one is the job of the log cleaner, a pool of threads inside the broker, and it runs on its own schedule under rules you set per topic.

Clean, dirty and active

The cleaner sees each partition as 3 regions. The active segment, where new records are appended, is never touched. Everything before it is either clean, meaning the cleaner has already passed over it and it holds at most 1 value per key, or dirty, meaning records have arrived since the last pass. The boundary is 1 offset per partition, saved in a file named cleaner-offset-checkpoint in each log directory, so it survives a restart.

The dirty ratio decides whether a pass is worth it

Rewriting segments costs a full read and a full write of their bytes, so the cleaner runs on a partition only once its dirty region has grown enough to pay for that. The measure is the dirty ratio: dirty bytes divided by clean plus dirty bytes, with the active segment left out. Say the clean segment in the table holds 100 MB and the 2 dirty segments hold 60 MB and 40 MB. The ratio is 100 divided by 200, which is 0.5. The topic setting min.cleanable.dirty.ratio defaults to 0.5, so this partition qualifies by a hair. When a cleaner thread wakes, it looks across every compacted partition on the broker, takes the eligible one with the highest ratio, cleans it, then looks again; if nothing qualifies it sleeps for log.cleaner.backoff.ms, 15 seconds by default.

def cleanable(clean_bytes, dirty_bytes, min_ratio=0.5):
    total = clean_bytes + dirty_bytes
    if total == 0:
        return False
    return dirty_bytes / total >= min_ratio

print(cleanable(100_000_000, 100_000_000))
print(cleanable(100_000_000, 20_000_000))

Lag settings bound the timing from both sides

When a pass may happen is bounded by 2 topic settings. min.compaction.lag.ms, default 0, keeps records out of the dirty region until their timestamps are that old: a reader that needs every change from the last 10 minutes sets it to 600000 and knows nothing younger is deduplicated. The cleaner stops at the first segment whose largest timestamp is younger than the lag, so 1 record stamped a year ahead stalls its segment and every later one. max.compaction.lag.ms, with a default so large it never fires, forces the opposite. Once the oldest dirty record is older than it, the partition becomes eligible whatever its ratio, so a topic that receives a trickle of writes still compacts inside a known window. A partition with a ratio of 0.1 and a 2 day old dirty record is cleaned because of the max lag; a partition with a ratio of 0.9 whose dirty records are all 3 minutes old waits because of the min lag.

What one pass does

A pass has 2 phases. First the cleaner reads the dirty region into an offset map that holds, for each key, the highest offset at which that key appears. The map lives in memory sized by log.cleaner.dedupe.buffer.size, 128 MB by default; when the dirty region holds more keys than fit, the pass covers only the prefix that fits and the rest waits for the next pass. Second it reads the clean and dirty segments from the start, copies into a new file each record whose offset is the one in the map for its key, and swaps the new file in for the group it replaced. Offsets never change. A reader positioned at offset 1500 resumes at 1500 even if 1500 itself was removed: the fetch returns the first surviving record after it.

A tombstone survives the pass that first reads it. That pass writes a deletion horizon beside it, and a later pass, once delete.retention.ms has elapsed past that horizon, drops it.

Try it. The drill below hands you 1 partition and the settings above; strike the records the cleaner has removed by the clock it shows.

About this lesson

This section is part of the The Log: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.