Timestamps from the Future
Concepts covered: Record timestamps
Time retention deletes a closed segment once it is old enough, and the broker decides age with the segment's largest timestamp: a segment is eligible when now minus its largest timestamp exceeds retention.ms, checked every log.retention.check.interval.ms, 5 minutes by default. Under CreateTime that largest timestamp is whatever the most forward-running producer clock claimed. One record stamped 3 years ahead raises its segment's largest timestamp by 3 years, and the segment stays on disk until the real clock catches up to that stamp plus the retention.
| segment | offsets | largest timestamp | age at 12 Mar 2027 09:00 | deleted with retention.ms = 7 days? |
|---|---|---|---|---|
| 0 | 0 to 9999 | 28 Feb 2027 23:58 | 11d 9h | yes |
| 10000 | 10000 to 19999 | 1 Mar 2027 23:59 | 10d 9h | yes |
| 20000 | 20000 to 29999 | 14 Mar 2030 11:20 | 3 years in the future | no, until 21 Mar 2030 |
| 30000 | 30000 to 39999 | 3 Mar 2027 23:57 | 8d 9h | no, stuck behind 20000 |
| 40000 | 40000 and up | 12 Mar 2027 08:59 | active | never while active |
Deletion walks the segments from the front and stops at the first one that is not old enough. Segment 20000 is stamped 3 years in the future, so it is never old enough, and segment 30000 behind it, which on its own is 8 days old and past the 7 day retention, is kept as well. Every segment appended after the future record is kept with it. One record of a few hundred bytes has turned a 7 day topic into one that keeps everything, and the disk fills at the partition's full write rate until somebody notices. Size retention, retention.bytes, still applies, because it counts bytes rather than reading stamps.
The guard, and when it is loosened
The broker refuses records that claim a time too far from its own clock, and 2 topic settings draw the window: message.timestamp.after.max.ms, default 3600000, 1 hour, rejects a record stamped further ahead than that, and message.timestamp.before.max.ms, with a default so large it never fires, would reject one stamped further behind. A rejected record fails the write with InvalidTimestampException, so the producer sees it and the log stays clean. With the defaults, the worst a future stamp can do is hold a segment 1 hour longer. The damage in the table needs the after window widened, which teams do when applications stamp records with a scheduled time, such as a reminder due next month, and then forget that retention reads the same field. Both settings apply only under CreateTime; LogAppendTime makes the broker's clock the stamp, so there is nothing to check.
message.timestamp.after.max.ms=3600000
message.timestamp.before.max.ms=9223372036854775807Records from the past expire early
The before window defaults to open because replays are routine: a producer re-reads 3 days of events from a file and writes them with their original timestamps. Under CreateTime those stamps stay, and if the topic's retention is 24 hours, every segment the replay fills is already older than the retention the moment it closes. The next retention check, within 5 minutes, deletes it. A reader that is 10 minutes behind the replay finds the offsets it wanted gone and starts again from the earliest offset the log still has. The replay looked like it worked, and the data was in the log for less time than it took to write it. The fixes are to raise retention.ms for the replay's duration, or to run the replay into a topic set to LogAppendTime, or to keep the original time inside the value and let the record's stamp be the time of the replay.
Rolling and lookup trust the stamp too
The same field feeds 2 more mechanisms. Time-based rolling measures from a segment's first stamp, so a segment that opens with a future-stamped record never rolls on segment.ms while new stamps trail it; it rolls on segment.bytes or waits. The time index records only stamps larger than any before them in the segment, so after the future record at offset 20000 no entry is added, and a lookup for any time from 2 Mar 2027 to 2030 picks the segment holding 20000, scans to it, and returns 20000. A reader that asks for 'everything since yesterday' is handed more than 10 days of data in place of 1.
LogAppendTime would not repair this: it stamps only later appends, and segment 20000 keeps its 2030 stamp. Set message.timestamp.after.max.ms back to 1 hour, then run kafka-delete-records.sh --bootstrap-server broker1:9092 --offset-json-file delete.json, the file naming partition 0 at offset 30000. The log's first offset moves to 30000, and the next check deletes segment 20000, whose other records were already past retention, with segment 30000 behind it, so lookups read real stamps again. A compact-only topic refuses the command; there, or where current records share the bad segment, copy the records into a new LogAppendTime topic.
The drill below gives you a partition with 1 future stamp in it; strike the records retention has deleted at the clock shown.
About this lesson
This section is part of the The Log: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.