Segments on Disk

The beginner lesson drew a partition as one tape of records numbered from 0. On the broker, the server that stores the partition, that tape is a directory. A topic orders with 3 partitions puts 3 directories under the broker's log.dirs path, named orders-0, orders-1 and orders-2. Inside each directory the records sit in segments, and a segment is a small group of files named by the offset of the first record it holds. The listing below is orders-0 after 2.4 million records of about 1 KiB each.

What orders-0 holds on disk

Each .log name is the base offset of its segment, padded with zeros so the names sort as text in offset order. That number lets the broker pick the right file for any offset without opening one: to serve a read at 1,500,000 it takes the largest base offset at or below the target, which is 1000000, and goes to that segment. Offsets 0 to 999,999 live in the first file, and a reader positioned at 1,500,000 never touches it.

Finding one offset inside a 1 GiB file

The second file, .index, maps offsets to byte positions inside the .log, and it is sparse: the broker adds an entry after every few kilobytes of appended data, not after every record. A lookup for 1,500,000 binary searches the index for the nearest entry at or below the target, seeks to that byte position in the .log, and scans forward a few kilobytes until it reaches the record. The .timeindex maps a timestamp to the offset of the first record at or after it, so a reader that asks for records since 09:00 is answered with an offset and then served through the .index like any other read. Neither index is the data: if one is missing or damaged the broker rebuilds it from the .log when it starts.

• 20
• 4,096 bytes
• 10 MiB

The active segment

Only the newest segment is written, and it is called the active segment. Every append goes to the end of its .log, and its 2 index files grow with it. When the active segment reaches segment.bytes, 1 GiB by default, or has been open for segment.ms, 7 days by default, the broker rolls: it closes the 3 files and opens a new segment whose base offset is the next offset to be written. A closed segment is never written again. Nothing is appended to it, and no record inside it is changed in place; the only things that ever happen to it are deletion as a whole and, on a compacted topic, being rewritten into a new file.

segment.bytes=1073741824
segment.ms=604800000

Segment size sets how much a partition can delete at once

Because a closed segment is a sealed unit, the segment is the smallest amount of data a partition can give back. A partition with 1 GiB segments cannot drop 10 MiB of old records; it drops 1 GiB or nothing. How big a segment gets before it rolls depends on traffic, because the 2 settings race and whichever is reached first wins.

A busy partition fills segments by size, so its files are uniform and each holds a narrow window of time. A quiet partition holds a whole week in one file, and that file is the unit it can expire. The next section's timing arithmetic depends on knowing which row of this table a topic sits in.

About this lesson

This section is part of the The Log: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.