Two Kinds of Timestamp

Concepts covered: Record timestamps

Every record carries 1 timestamp: a 64-bit count of milliseconds since the epoch, stored in the record itself on disk. Whose clock it came from is the topic setting message.timestamp.type, which has 2 values. CreateTime, the default, keeps what the producer sent: the producer's own clock at the moment the record was built, unless the application set an explicit value such as the time a sensor took its reading. LogAppendTime discards that and stamps the record with the broker's clock as the record is appended to the log.

CreateTime (default)

• Stamped by the producer, or by the application if it sets one
• Can go backwards between neighbouring records
• Preserves when the event happened
• Trusts every producer's clock

LogAppendTime

• Stamped by the broker as it writes the record
• Never decreases within a partition
• Records when the broker received it, and nothing else
• Ignores the producer's clock and any value the application set

message.timestamp.type=LogAppendTime

The choice decides what the number means. Under CreateTime, the timestamp at offset 122 can be earlier than the one at 121, because 2 producers with clocks 4 seconds apart, or 1 producer resending after a network stall, append in an order their clocks did not agree on. Under LogAppendTime the stamps in a partition never decrease, because 1 clock assigns them in append order. The cost is that the moment the event happened is gone unless the application carries it inside the value. LogAppendTime suits topics where only arrival order and arrival time matter, such as feeds from devices whose clocks nobody maintains.

The time index

Beside each segment's offset index sits a second file ending in .timeindex, a list of (timestamp, offset) pairs. The broker appends an entry about every index.interval.bytes of data, 4096 bytes by default, and only if the record's timestamp is larger than every timestamp already in that segment's index. The entries therefore increase even on a CreateTime topic whose records do not.

Finding an offset by time

A lookup asks for the earliest offset whose record is stamped at or after a time T. The broker first chooses a segment: the earliest one whose largest timestamp is at or after T. Within it, a binary search over the time index finds the entry with the largest timestamp not after T, and the broker scans forward from that entry's offset, reading records, until one is stamped at or after T. Take T = 10:00:05 in the tables. The entry is (10:00:00, 120). The scan reads 120, 121, 122 and 123, and stops at 123 because 10:00:09 is the first stamp at or after 10:00:05. Offset 124, stamped 10:00:07, comes after T in value but sits after 123 in position, so a reader that starts at 123 receives it. That is the guarantee a lookup gives: every record stamped at or after T lies at or after the returned offset, while records stamped before T may still lie after it.

Under LogAppendTime the scan after the binary search is short, because stamps increase with offsets, and the answer is exact to the broker's clock. Under CreateTime the answer is only as good as the producers' clocks: a producer running 10 minutes slow writes records that a lookup for the real time places at the wrong offset, and no index can repair that after the fact.

On the command line, kafka-get-offsets.sh --bootstrap-server broker1:9092 --topic clicks --time 1762250405000 prints that offset for every partition of the topic, and a consumer gets the same answer from its offsetsForTimes call. In place of a time you can pass 3 words: earliest returns the first offset still in the log, latest the offset after the last record, and max-timestamp the offset of the record with the largest timestamp.

The animation sends the same 6 records from offset 0, each tile showing its stamp's seconds, so :09 sits at offset 3. One run starts a reader at offset 0; the other starts it at offset 3, the lookup's answer for 10:00:05. Watch which records each run receives.

Do

• Set the timestamp explicitly when the event time is known, so CreateTime means event time
• Use LogAppendTime on topics where producer clocks are untrusted
• Run kafka-get-offsets.sh with --time before replaying from a date

Avoid

• Assume CreateTime stamps increase along a partition
• Read a lookup's result as 'only records after T'
• Change message.timestamp.type on a live topic without telling its readers

About this lesson

This section is part of the The Log: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.