The Log: Intermediate

The beginner lesson drew a partition as one tape of records numbered from 0. On the broker, the server that stores the partition, that tape is a directory. A topic orders with 3 partitions puts 3 directories under the broker's log.dirs path, named orders-0, orders-1 and orders-2. Inside each directory the records sit in segments, and a segment is a small group of files named by the offset of the first record it holds. The listing below is orders-0 after 2.4 million records of about 1 KiB each.

Each .log name is the base offset of its segment, padded with zeros so the names sort as text in offset order. That number lets the broker pick the right file for any offset without opening one: to serve a read at 1,500,000 it takes the largest base offset at or below the target, which is 1000000, and goes to that segment. Offsets 0 to 999,999 live in the first file, and a reader positioned at 1,500,000 never touches it.

The second file, .index, maps offsets to byte positions inside the .log, and it is sparse: the broker adds an entry after every few kilobytes of appended data, not after every record. A lookup for 1,500,000 binary searches the index for the nearest entry at or below the target, seeks to that byte position in the .log, and scans forward a few kilobytes until it reaches the record. The .timeindex maps a timestamp to the offset of the first record at or after it, so a reader that asks for records since 09:00 is answered with an offset and then served through the .index like any other read. Neither index is the data: if one is missing or damaged the broker rebuilds it from the .log when it starts.

Only the newest segment is written, and it is called the active segment. Every append goes to the end of its .log, and its 2 index files grow with it. When the active segment reaches segment.bytes, 1 GiB by default, or has been open for segment.ms, 7 days by default, the broker rolls: it closes the 3 files and opens a new segment whose base offset is the next offset to be written. A closed segment is never written again. Nothing is appended to it, and no record inside it is changed in place; the only things that ever happen to it are deletion as a whole and, on a compacted topic, being rewritten into a new file.

Retention is the broker deleting old segments on a schedule, and 2 topic settings drive it. retention.ms is how long a record may stay, 7 days by default. retention.bytes caps the bytes a partition may hold, and its default of -1 means no cap. Both are enforced per partition, not per topic: a topic with 12 partitions and retention.bytes of 1 GiB holds about 12 GiB, plus up to 1 segment per partition on top, as the size rule below shows. Reading plays no part at all. A record fetched 1,000 times and a record never fetched expire at the same moment.

A topic-level retention.ms overrides the broker-wide log.retention.ms and log.retention.hours, so set it on the topic, where the number sits next to the data it governs.

Every log.retention.check.interval.ms, 5 minutes by default, a background thread visits each partition and walks its segments from oldest to newest. For each one it takes the largest timestamp the broker recorded among the records inside. If the clock minus that timestamp is more than retention.ms, the segment is deleted whole. The walk stops at the first segment young enough to keep, since later segments hold later records. If the walk reaches the active segment and it is past the window too, the broker rolls a fresh empty segment first and then deletes the old one, so a partition always keeps at least 1 segment and an empty one is never deleted.

The second segment keeps its records from Monday 08:01, which are past the window, because the rule reads only the newest timestamp in each segment and judges the whole segment by it. Those records leave on Saturday 10:00, or at the first check after it, when the segment's newest record turns 3 days old. The time rule alone never removes a record before retention.ms, and it rarely removes one at retention.ms. When retention.bytes is also set, the size rule can take a segment sooner, whatever its age.

retention.bytes is checked against the partition's total bytes, active segment included. A partition that has written less than the cap keeps everything. Once it passes the cap, the thread removes the oldest segment only while the partition would still hold at least retention.bytes after the removal, so deletion stops at the last point at or above the cap and the partition can sit up to 1 segment over it. With a 2 GiB cap and 1 GiB segments, a partition holding 3 closed segments and 512 MiB in the active one has 3.5 GiB. Removing the oldest leaves 2.5 GiB, still over the cap, so it goes. Removing the next would leave 1.5 GiB, under the cap, so it stays. The partition settles at 2.5 GiB and will not shrink further until more is written.

When both settings are on, a segment goes as soon as either rule says so. Neither limit is exact: the time rule removes a segment only after its newest record is older than retention.ms, and the size rule holds a partition at no more than retention.bytes plus 1 segment.

A reader whose position sits inside a deleted segment cannot resume there: the records it was about to fetch are gone, and the earliest offset the partition can serve has moved up to the base offset of the oldest surviving segment. Lowering retention.ms on a live topic takes effect at the next check, so look at where the slowest reader stands before changing it.

Some topics carry state rather than events. A topic of customer addresses keys each record by customer id and carries the current address as the value. The 5th address c-7 sent makes the earlier 4 worthless, but the delete policy would also drop the 5th once it turned 7 days old, and a reader building a table of addresses would then find c-7 missing. cleanup.policy=compact changes what a partition keeps: for each key, the record with the highest offset is never removed, and older records with the same key may be. A reader starting at offset 0 receives every key's latest value, however old it is, and the partition's size tracks the number of distinct keys rather than the number of writes.

The pass left 4 records, 1 per key, and each kept the offset it was written at. Offsets 0 to 3 no longer exist, and a reader that asks for offset 1 is handed the record at 4, the next offset the partition has. The survivors stay in their original order, so c-2 at 4 still comes before c-7 at 5. Compaction never creates or moves a record; it only leaves some out when it rewrites a segment.

Cleaning is per partition, so the promise of 1 latest record per key holds only when every record for a key lands in the same partition, which the key itself arranges and the last section returns to.

The policy is set per topic, at creation with kafka-topics.sh --create --bootstrap-server localhost:9092 --topic customer-addresses --partitions 6 --config cleanup.policy=compact or later with kafka-configs.sh. A topic may carry both policies, cleanup.policy=compact,delete: the cleaner keeps the latest value per key, and the retention thread still removes segments whose newest record is older than retention.ms, so a key that stops being written vanishes after the window.

Suppose the cleaner removed the tombstone in the same pass that removed the old values. A reader that was halfway through the partition, or offline for the night, would reach the end and never learn that c-7 was deleted; its table would carry a stale address with nothing left in the topic to correct it. So the tombstone is kept after the values are gone, for delete.retention.ms, 24 hours by default, measured from the moment the cleaner first processed the segment that holds it. A later pass after that time removes it. A reader that reads through within the window sees the tombstone and deletes the key. A reader that arrives after it sees no record for c-7 at all, and that is also correct: the key is absent.

Each row ends in the same state for a reader that starts from offset 0, which is the point of the design. The window matters for a reader that already holds c-7 from an earlier read and comes back later: if it returns before Wednesday 03:05 it sees the tombstone and deletes the key; if it returns after, nothing in the topic mentions c-7, and the stale row survives in its table.

A reader that skips null values, which some deserializers tempt it to do by failing on an empty payload, keeps every deleted key. Processing in offset order makes the rest work without special cases: in the Monday row above the tombstone sits in the active segment behind 3 live values the cleaner has not touched, and a reader applying each record in turn still finishes with c-7 removed.

A record with a key goes to the partition at hash(key) mod partition count, and the Java client computes that hash with murmur2. With 3 partitions, cust-7 maps to the same partition every time, which is what lets a compacted partition hold the key's latest value and lets a reader see the key's records in order. Change the count to 4 and hash mod 4 sends cust-7 somewhere else. The animation sends the same 8 records, 4 keys, to 2 topics side by side: orders with 3 partitions and orders-v2 with 4, or with 6 at the switch.

Follow 1 key across the 2 topics. In orders its 2 records land together, and in orders-v2 they land together again, though some keys sit on a different partition number than they do in orders. Either way a key's records stay together; only the directory that holds them changes. If orders were a topic's past and orders-v2 its future at a higher count, a key that moved would be split across 2 directories.

kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic orders --partitions 4 creates 1 new empty directory on a broker and tells producers that there are now 4 targets. No existing record moves, because the segments are closed files and compaction and retention are the only things that rewrite them. A key's history now lives in 2 places: its past in the partition it used to hash to, its future in the partition it hashes to now. For that key, 2 things break. Its order across the change is gone, since a reader working 2 partitions can be anywhere in one when it reads the other. And on a compacted topic, the old partition keeps the key's old value forever; a tombstone sent later goes to the new partition and removes nothing in the old one, so a reader rebuilding a table from offset 0 can finish with the stale value, depending on which partition it reads last.