Retention by Time and Size

Retention is the broker deleting old segments on a schedule, and 2 topic settings drive it. retention.ms is how long a record may stay, 7 days by default. retention.bytes caps the bytes a partition may hold, and its default of -1 means no cap. Both are enforced per partition, not per topic: a topic with 12 partitions and retention.bytes of 1 GiB holds about 12 GiB, plus up to 1 segment per partition on top, as the size rule below shows. Reading plays no part at all. A record fetched 1,000 times and a record never fetched expire at the same moment.

A topic-level retention.ms overrides the broker-wide log.retention.ms and log.retention.hours, so set it on the topic, where the number sits next to the data it governs.

The time rule reads one timestamp per segment

Every log.retention.check.interval.ms, 5 minutes by default, a background thread visits each partition and walks its segments from oldest to newest. For each one it takes the largest timestamp the broker recorded among the records inside. If the clock minus that timestamp is more than retention.ms, the segment is deleted whole. The walk stops at the first segment young enough to keep, since later segments hold later records. If the walk reaches the active segment and it is past the window too, the broker rolls a fresh empty segment first and then deletes the old one, so a partition always keeps at least 1 segment and an empty one is never deleted.

The second segment keeps its records from Monday 08:01, which are past the window, because the rule reads only the newest timestamp in each segment and judges the whole segment by it. Those records leave on Saturday 10:00, or at the first check after it, when the segment's newest record turns 3 days old. The time rule alone never removes a record before retention.ms, and it rarely removes one at retention.ms. When retention.bytes is also set, the size rule can take a segment sooner, whatever its age.

Four reasons the window is never exact

Where the gap between retention.ms and the real departure comes from

• The segment rule: A record leaves when the newest record in its segment is old enough. In a 1 GiB segment that can be hours or days after the record's own timestamp passed the window.
• The check interval: A segment that becomes old enough at 09:01 waits for the check at 09:05, and longer if the interval has been raised on the broker.
• The record's timestamp: Age is measured from the timestamp the record carries, so the window is counted from the record's stamp rather than from the moment the broker wrote it.
• The delete delay: A deleted segment is first renamed with a .deleted suffix and its files are unlinked after file.delete.delay.ms, 1 minute by default, so the disk space comes back a minute after the check.

The size rule is a ceiling that overshoots

retention.bytes is checked against the partition's total bytes, active segment included. A partition that has written less than the cap keeps everything. Once it passes the cap, the thread removes the oldest segment only while the partition would still hold at least retention.bytes after the removal, so deletion stops at the last point at or above the cap and the partition can sit up to 1 segment over it. With a 2 GiB cap and 1 GiB segments, a partition holding 3 closed segments and 512 MiB in the active one has 3.5 GiB. Removing the oldest leaves 2.5 GiB, still over the cap, so it goes. Removing the next would leave 1.5 GiB, under the cap, so it stays. The partition settles at 2.5 GiB and will not shrink further until more is written.

When both settings are on, a segment goes as soon as either rule says so. Neither limit is exact: the time rule removes a segment only after its newest record is older than retention.ms, and the size rule holds a partition at no more than retention.bytes plus 1 segment.

retention.ms=259200000
retention.bytes=2147483648
segment.bytes=268435456

A reader whose position sits inside a deleted segment cannot resume there: the records it was about to fetch are gone, and the earliest offset the partition can serve has moved up to the base offset of the oldest surviving segment. Lowering retention.ms on a live topic takes effect at the next check, so look at where the slowest reader stands before changing it.

Do

• Size retention.bytes per partition and multiply by the partition count for the topic.
• Give short-retention topics smaller segments so expiry tracks the window.
• Leave headroom in retention.ms for the longest outage a reader may have.

Avoid

• Expect disk to come back at the moment retention.ms passes.
• Count on reading to free space; only the clock and the byte cap do.
• Lower retention.ms on a live topic without checking the slowest reader.

About this lesson

This section is part of the The Log: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.