In-Sync Replicas
Concepts covered: In-sync replicas
Caught up is measured in seconds
A follower that copies continuously is never more than a few milliseconds behind, but a follower on a sick disk, or behind a saturated network link, can fall far behind, and the cluster needs a crisp rule for which followers still count. The rule defines the in-sync replica set, written ISR. A follower is in the ISR while it has caught up to the leader's end of log at some point in the last replica.lag.time.max.ms milliseconds, 30000 by default. The leader of a partition is always in its own ISR.
# broker setting, default shown
replica.lag.time.max.ms=30000Read the rule carefully, because it is a rule about time. A follower that is 2,000 records behind at this instant is in sync if it reached the leader's end 10 seconds ago, which happens during a burst: the leader took a flood of records, and the follower is fetching through it. A follower that is 1 record behind is out of sync if it has reached the leader's end at no point in the last 30 seconds, which happens when the follower is stuck. The leader keeps a clock for each follower, and when 30 seconds pass without that follower reaching the end, it drops the follower from the ISR.
Kafka once judged lag by a count of records, and dropped that rule years ago: a burst from a producer could push a healthy follower past the count and out of the set through no fault of its own, and a quiet partition could hide a stuck follower that was only 3 records behind. Time catches both cases, because a healthy follower reaches the end many times a second whatever the volume.
The ISR lists the safe successors
The ISR answers 1 question: which replicas could take over as leader without losing records? A follower in the ISR has held everything the leader held within the last 30 seconds and is fetching the rest. A follower dropped from the ISR may be missing hours of records. When section 4's broker dies, the cluster picks the new leader from the ISR and only from the ISR, so keeping the set accurate is what makes a failover safe. While a follower is out, the partition still takes writes and the leader still confirms them; the cost is 1 fewer safe candidate, so a second failure in that window hurts more. How many in-sync copies a write must reach before the leader confirms it is a setting the next replication lesson covers.
A follower falls behind and comes back
| Time | Leader log end | Broker 3 log end | Broker 3 last reached the end | ISR |
|---|---|---|---|---|
| 09:00:00 | 500 | 500 | 09:00:00 | 1,2,3 |
| 09:00:10 | 900 | 620 | 09:00:04 | 1,2,3 |
| 09:00:35 | 1900 | 700 | 09:00:04 | 1,2 |
| 09:01:20 | 3700 | 3700 | 09:01:20 | 1,2,3 |
At 09:00:04 broker 3's disk started stalling. It kept answering, slowly, so at 09:00:10 it was 280 records behind and still in the set, because its last full catch-up was 6 seconds old. At 09:00:34 that catch-up turned 30 seconds old and the leader removed broker 3 from the ISR; writes to the partition continued through brokers 1 and 2. Broker 3's fetch loop kept running without any restart, the disk recovered, and at 09:01:20 it reached offset 3700, the leader's end, so the leader added it back. Shrinking and expanding the ISR is routine and automatic; what an operator watches for is a shrink that does not undo itself.
- 3 brokers
- orders: 1 partition, replication factor 3
- Healthy followers: Both followers reach the leader's end within milliseconds of each record.
- Broker 3 stalls: Broker 3's fetches take 6 s each from 1.5 s, it is dropped from the ISR after 3 s without reaching the end, and it rejoins after recovering at 9 s.
Broker 1 is the leader of orders-0; brokers 2 and 3 follow. In the second variant broker 3's fetches crawl from 1.5 s, it leaves the ISR, and it rejoins once its log reaches the leader's end. The lag limit is 3 s here so the scene fits.
Running --describe at 09:00:35 prints Isr: 1,2 against Replicas: 1,2,3: 1 replica is assigned and not caught up. The lag rule covers a follower that is alive and stuck; a broker that stops altogether leaves every ISR sooner, about 9 seconds after it falls silent, through the session timeout section 4 introduces.
About this lesson
This section is part of the Replication: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.