Followers That Fall Behind
Concepts covered: In-sync replicas
In sync is a statement about time. A follower is in sync when it has fetched up to the leader's log end offset at least once in the last replica.lag.time.max.ms milliseconds, 30,000 by default. The leader records the moment each follower last caught up fully, and a follower whose moment is older than 30 seconds is removed from the in-sync set. The number of records behind does not enter the rule: a follower 5,000 records behind a busy leader that caught up 2 seconds ago is in sync, and a follower 1 record behind that has not reached the end for 31 seconds is out.
Time works where a count would not: a partition taking 10,000 records a second always has a follower hundreds of records behind mid-fetch, so a count threshold would flap on every burst.
Timeline of a shrink
| Clock | What happens | In-sync set | High watermark |
|---|---|---|---|
| 0 s | Broker 3 enters a long garbage collection pause and stops fetching | 1, 2, 3 | stuck at 1,200 |
| 0 to 30 s | Leader appends 3,000 records; broker 2 copies them all; broker 3 copies none | 1, 2, 3 | stuck at 1,200 |
| 30 s | Leader's check finds broker 3 last caught up 30 s ago and removes it | 1, 2 | jumps to 4,200 |
| 30 s | Leader sends the new set to the controller, which records it in the metadata log | 1, 2 | 4,200 |
| 45 s | Broker 3 wakes and resumes fetching from offset 1,200 | 1, 2 | advancing |
| 52 s | Broker 3 reaches the high watermark and is added back | 1, 2, 3 | advancing |
For those 30 seconds every record the leader accepted sat above the high watermark, confirmed to no acks=all producer and readable by nobody. Removing the follower unblocks the partition: once the set is brokers 1 and 2, broker 2's log end offset sets the mark, which jumps 3,000 offsets and hands stalled readers a burst. With a minimum of 2, the partition still accepts fully replicated writes, 1 broker away from refusing them.
The leader runs that check every replica.lag.time.max.ms divided by 2, so a follower leaves between 30 and 45 seconds after it stalls. The leader proposes the smaller set to the controller and keeps using the old one until the controller has recorded the change, so a slow controller keeps the mark pinned a little longer.
What makes a follower slow
- A disk that cannot keep up: The follower appends slower than the leader, falls behind during bursts and never quite reaches the end.
- A saturated or partitioned network link: Cross-zone links are the common case: replication traffic competes with everything else on them.
- A long garbage collection pause: Nothing is fetched for seconds; a pause past 30 s costs the broker its place in every partition it follows.
- Too many partitions on 1 broker: The follower rotates through thousands of partitions and each one waits its turn.
- A restart: A broker that stops is removed as soon as the controller notices it is gone, then catches up on everything it missed.
Coming back
A follower rejoins once the offset it requests reaches the current high watermark, which can be short of the leader's log end. The leader then proposes adding it back, and the set grows when the controller records the change. From then on the mark advances only as fast as that follower copies. A follower that is still unhealthy, catching up in a quiet moment and falling behind on the next burst, flaps in and out, and each time it is in, the mark can wait on it for up to 30 seconds.
kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions lists every partition whose in-sync set is smaller than its replica list, and prints nothing on a healthy cluster. The broker metric UnderReplicatedPartitions counts the same per broker, and UnderMinIsrPartitionCount counts the partitions below their minimum, which are refusing acks=all writes. A few under-replicated partitions for a minute during a rolling restart is normal; a list that stays put points at a follower that cannot keep up.
- 3 brokers
- clicks: 1 partition, replication factor 3
- Slow fetches: Removed when the lag timer runs out
- Broker restart: Removed as soon as the controller sees it gone
Broker 2 drops out of the in-sync set and rejoins at the watermark: slowed in one variant, stopped and restarted in the other.
In both variants the watermark sticks while broker 2 still counts as in sync and jumps when it leaves. A stopped broker loses its place as soon as the controller sees it gone; a slow one keeps it until the lag timer, shortened to 1.5 seconds here, runs out, because until then a slow fetcher looks like a healthy one.
About this lesson
This section is part of the Replication: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.