The High Watermark
Concepts covered: In-sync replicas
A producer sends a record to the partition's leader on broker 1. The leader appends it at offset 7 and the record now exists on exactly 1 disk in the cluster. A consumer fetching from offset 7 at that instant gets back an empty response. The leader holds the record back until every in-sync follower has copied it, and the offset that marks how far that copying has reached is the high watermark.
Two offsets on every replica
Every replica tracks 2 positions. The log end offset is the offset of the next record it will append, so a replica whose last record sits at offset 7 has a log end offset of 8. The high watermark is the offset up to which every replica in the in-sync set holds the data. On the leader, the high watermark is the smallest log end offset among the in-sync replicas. A record below the high watermark is committed: it survives the loss of any single in-sync replica, because every one of them has it.
Take a partition with replication factor 3: the leader on broker 1, followers on brokers 2 and 3, all 3 in sync, and 7 records already committed at offsets 0 through 6. Call the follower on broker 2 follower A and the one on broker 3 follower B. The table follows 1 new record from its append to the moment a reader can have it.
| Moment | Leader log end offset | Follower A next fetch | Follower B next fetch | High watermark | A reader can see |
|---|---|---|---|---|---|
| Record 7 appended on the leader | 8 | 7 | 7 | 7 | offsets 0 to 6 |
| A's fetch for offset 7 returns record 7 | 8 | 7 (writing it) | 7 | 7 | offsets 0 to 6 |
| A asks for offset 8, B asks for offset 7 | 8 | 8 | 7 | 7 | offsets 0 to 6 |
| B asks for offset 8 | 8 | 8 | 8 | 8 | offsets 0 to 7 |
The leader learns what a follower holds only from its fetch requests: a follower requesting offset 8 is proving it has written everything through offset 7. So a record needs 2 round trips before a reader can have it: the fetch that carries it to the follower, then the next fetch, whose offset tells the leader it landed. On a healthy partition that takes a few milliseconds. When 1 in-sync follower struggles, the wait stretches to that follower's pace, because the slowest member sets the smallest log end offset.
What the reader is allowed to see
A consumer's fetch names an offset, and the leader answers with records from there up to the high watermark and no further, plus the mark itself. A consumer that asks for offset 7 while the mark is still 7 receives 0 records, waits, and asks again. Every fetch response to a follower carries the mark too, and each follower raises its own copy to match, so a follower that takes over as leader already knows which records readers may have.
Reading the mark from the command line
kafka-get-offsets.sh --bootstrap-server broker1:9092 --topic orders prints 1 line per partition in the form orders:0:8, and the number is the high watermark, the first offset no reader has been given. When it stops moving while the producer keeps sending, a follower is holding it, and the next 2 sections show what the leader does about it.
- 3 brokers
- orders: 1 partition, replication factor 3
- Followers keep up: The watermark trails the leader by a fetch
- Broker 3 fetches slowly: Readers stall at the watermark until it catches up
Broker 1 leads. In the slow variant broker 3 fetches late and the consumer stalls at the watermark until it catches up.
In the slow variant the leader keeps appending while the consumer's position freezes at the watermark. Broker 3 stays in the in-sync set throughout, since its stall is far shorter than 30 seconds, so the mark is pinned to its progress. When broker 3 catches up, the mark jumps and the consumer takes the held records in a burst.
About this lesson
This section is part of the Replication: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.