Replication: Intermediate

Every replica tracks 2 positions. The log end offset is the offset of the next record it will append, so a replica whose last record sits at offset 7 has a log end offset of 8. The high watermark is the offset up to which every replica in the in-sync set holds the data. On the leader, the high watermark is the smallest log end offset among the in-sync replicas. A record below the high watermark is committed: it survives the loss of any single in-sync replica, because every one of them has it.

kafka-get-offsets.sh --bootstrap-server broker1:9092 --topic orders prints 1 line per partition in the form orders:0:8, and the number is the high watermark, the first offset no reader has been given. When it stops moving while the producer keeps sending, a follower is holding it, and the next 2 sections show what the leader does about it.

Replication factor 3 promises 3 copies of every record, but the promise only holds while 3 brokers are up. With min.insync.replicas at its default of 1, a partition whose 2 followers have both dropped out of the in-sync set keeps accepting writes with the leader as the only replica, and the leader confirms each record as fully replicated because, among the in-sync replicas, it is. If that broker's disk fails before the followers return, those records are gone. min.insync.replicas sets the floor: the number of in-sync replicas a partition must have before it accepts a write that asked for full replication.

A producer chooses how much confirmation it wants through its acks setting. With acks=all, the leader confirms a record only after every in-sync replica has it, which in high watermark terms means after the mark has passed the record. That is the write min.insync.replicas guards. When the in-sync set is smaller than the minimum, the leader answers an acks=all send with the error NotEnoughReplicas and appends nothing. A producer sending with acks=1 asked only for the leader's confirmation, and the leader keeps accepting those writes with 1 copy, minimum or not. The setting protects producers that asked for safety and does nothing for ones that did not.

With the eligible leader replicas feature, on by default on clusters newly formatted with a recent 4.x release and switched on by an operator after an upgrade, the rule also reaches the high watermark: while the in-sync set is below the minimum, the leader stops advancing the mark. Records that arrived with acks=1 during the shortage stay unreadable until enough replicas are back in sync, so no reader acts on a record only 1 disk holds.

The setting lives on the topic, with a broker-wide default for topics that do not set their own. The usual pairing is replication factor 3 with a minimum of 2. kafka-configs.sh --bootstrap-server broker1:9092 --alter --entity-type topics --entity-name payments --add-config min.insync.replicas=2 raises it on 1 topic at once. The broker properties below set the floor for every topic without its own value, existing topics included, once each broker restarts with them; default.replication.factor shapes only topics created afterwards.

With 1 broker down the in-sync set holds 2 replicas, the minimum, and both variants keep writing. The second kill drops the set to 1. In the acks=all variant every send from that point comes back with NotEnoughReplicas and the leader's log stops growing; in the acks=1 variant the log keeps growing on a single disk, which is the situation the setting exists to prevent.

In sync is a statement about time. A follower is in sync when it has fetched up to the leader's log end offset at least once in the last replica.lag.time.max.ms milliseconds, 30,000 by default. The leader records the moment each follower last caught up fully, and a follower whose moment is older than 30 seconds is removed from the in-sync set. The number of records behind does not enter the rule: a follower 5,000 records behind a busy leader that caught up 2 seconds ago is in sync, and a follower 1 record behind that has not reached the end for 31 seconds is out.

For those 30 seconds every record the leader accepted sat above the high watermark, confirmed to no acks=all producer and readable by nobody. Removing the follower unblocks the partition: once the set is brokers 1 and 2, broker 2's log end offset sets the mark, which jumps 3,000 offsets and hands stalled readers a burst. With a minimum of 2, the partition still accepts fully replicated writes, 1 broker away from refusing them.

The leader runs that check every replica.lag.time.max.ms divided by 2, so a follower leaves between 30 and 45 seconds after it stalls. The leader proposes the smaller set to the controller and keeps using the old one until the controller has recorded the change, so a slow controller keeps the mark pinned a little longer.

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions lists every partition whose in-sync set is smaller than its replica list, and prints nothing on a healthy cluster. The broker metric UnderReplicatedPartitions counts the same per broker, and UnderMinIsrPartitionCount counts the partitions below their minimum, which are refusing acks=all writes. A few under-replicated partitions for a minute during a rolling restart is normal; a list that stays put points at a follower that cannot keep up.

Broker 1 leads partition 0 of payments and stops answering. The active controller, the node that keeps the cluster's metadata, expects a heartbeat from every broker, and once broker.session.timeout.ms, 9,000 ms by default, passes with no heartbeat from broker 1, it marks broker 1 fenced, strips it out of every in-sync set, and picks a new leader for each partition broker 1 led. The election itself takes milliseconds and consults no producer or consumer; clients find the new leader when a request to the old one fails and they refresh their metadata.

A fenced broker leads nothing and holds no place in any in-sync set. A silent broker is fenced after the 9 second heartbeat timeout, well before the 30 second lag rule would have removed it as a slow follower.

The fields below are the ones kafka-topics.sh --describe prints for a partition. Broker 2 wins because it comes before broker 0 in the list and both are in sync.

Broker 1 may have held records above the high watermark that no follower had fetched: appended, confirmed to an acks=1 producer, read by nobody. Each election raises the partition's leader epoch, a counter every replica stores alongside its log and every request carries, so when broker 1 returns it is a follower with a stale epoch. It asks the new leader where the new epoch began, truncates its own log back to that offset, which discards the unreplicated records, and fetches forward from there. The acks=1 producer lost records it was told were stored, which is the risk it accepted, and no reader lost anything, because readers were held at the mark.

The in-sync set can be empty: brokers 2 and 0 die together, or 1 follower was already out of the set when the leader died. On its own, the in-sync rule then leaves the controller no safe candidate. The eligible leader replicas feature, on by default on clusters newly formatted with a recent 4.x release and switched on by an operator after an upgrade, adds a second list, which the describe output prints as Elr. A replica joins it when its own exit from the in-sync set takes the set below min.insync.replicas. The mark cannot advance while the set is short, so that replica still holds every committed record and the controller may elect it. A replica that has lost its data, such as one restarting onto a fresh disk, is taken off the list.

If the in-sync set and the eligible list are both empty, the partition goes offline: the describe output shows Leader: none, every read and write to it fails, and the partition waits for a replica from one of the 2 lists to come back. The controller would rather serve nothing than elect a replica that is missing committed records. An operator can override that and accept the loss; that choice has a lesson of its own.

Broker 1 rejoins as a follower while broker 2 keeps leading, so after a few restarts leadership piles up on whichever brokers stayed up. With auto.leader.rebalance.enable at its default of true, the controller checks every leader.imbalance.check.interval.seconds, 300 by default, and moves leadership back to the preferred replica on any broker whose share of non-preferred leaders exceeds 10 percent. kafka-leader-election.sh --bootstrap-server broker1:9092 --election-type preferred --all-topic-partitions does the same on demand, which is how an operator restores balance right after a rolling restart instead of waiting 5 minutes.

broker.rack is a label and nothing more; the broker does not check it against anything. The controller reads it when a topic is created or gains partitions and places each partition's replicas so that they spread across as many distinct racks as the replication factor allows, which with 3 racks and replication factor 3 means 1 replica per rack for every partition. It also rotates which broker holds the first replica, so leadership spreads evenly. The placement happens once, at creation: adding a rack label to a running broker changes nothing about partitions that already exist.

Every partition has 1 copy in each zone, and each broker leads exactly 1 partition and follows 2. With min.insync.replicas set to 2, the table also tells you what the cluster can survive.