Electing a New Leader
Concepts covered: Replication and leaders
Broker 1 leads partition 0 of payments and stops answering. The active controller, the node that keeps the cluster's metadata, expects a heartbeat from every broker, and once broker.session.timeout.ms, 9,000 ms by default, passes with no heartbeat from broker 1, it marks broker 1 fenced, strips it out of every in-sync set, and picks a new leader for each partition broker 1 led. The election itself takes milliseconds and consults no producer or consumer; clients find the new leader when a request to the old one fails and they refresh their metadata.
A fenced broker leads nothing and holds no place in any in-sync set. A silent broker is fenced after the 9 second heartbeat timeout, well before the 30 second lag rule would have removed it as a slow follower.
The rule
The controller walks the partition's replica list in order and picks the first replica that is both alive and in the in-sync set. The replica list is the one assigned to the partition, which only a reassignment changes, and its first entry is the preferred leader, so when nothing has gone wrong the preferred replica leads. The in-sync requirement is the heart of it: an in-sync replica holds every record below the high watermark, and the high watermark bounds everything any reader has seen, so a leader chosen from the in-sync set can never lose a record a reader already has.
The fields below are the ones kafka-topics.sh --describe prints for a partition. Broker 2 wins because it comes before broker 0 in the list and both are in sync.
| Moment | Leader | Replicas | Isr | Elr |
|---|---|---|---|---|
| Before | 1 | 1, 2, 0 | 1, 2, 0 | |
| Broker 1 fenced | 2 | 1, 2, 0 | 2, 0 | |
| Broker 1 back, catching up | 2 | 1, 2, 0 | 2, 0 | |
| Broker 1 reaches the mark | 2 | 1, 2, 0 | 2, 0, 1 | |
| Preferred leader election | 1 | 1, 2, 0 | 2, 0, 1 |
What happens to the old leader's extra records
Broker 1 may have held records above the high watermark that no follower had fetched: appended, confirmed to an acks=1 producer, read by nobody. Each election raises the partition's leader epoch, a counter every replica stores alongside its log and every request carries, so when broker 1 returns it is a follower with a stale epoch. It asks the new leader where the new epoch began, truncates its own log back to that offset, which discards the unreplicated records, and fetches forward from there. The acks=1 producer lost records it was told were stored, which is the risk it accepted, and no reader lost anything, because readers were held at the mark.
When nobody in sync is left
The in-sync set can be empty: brokers 2 and 0 die together, or 1 follower was already out of the set when the leader died. On its own, the in-sync rule then leaves the controller no safe candidate. The eligible leader replicas feature, on by default on clusters newly formatted with a recent 4.x release and switched on by an operator after an upgrade, adds a second list, which the describe output prints as Elr. A replica joins it when its own exit from the in-sync set takes the set below min.insync.replicas. The mark cannot advance while the set is short, so that replica still holds every committed record and the controller may elect it. A replica that has lost its data, such as one restarting onto a fresh disk, is taken off the list.
If the in-sync set and the eligible list are both empty, the partition goes offline: the describe output shows Leader: none, every read and write to it fails, and the partition waits for a replica from one of the 2 lists to come back. The controller would rather serve nothing than elect a replica that is missing committed records. An operator can override that and accept the loss; that choice has a lesson of its own.
Giving the leader back
Broker 1 rejoins as a follower while broker 2 keeps leading, so after a few restarts leadership piles up on whichever brokers stayed up. With auto.leader.rebalance.enable at its default of true, the controller checks every leader.imbalance.check.interval.seconds, 300 by default, and moves leadership back to the preferred replica on any broker whose share of non-preferred leaders exceeds 10 percent. kafka-leader-election.sh --bootstrap-server broker1:9092 --election-type preferred --all-topic-partitions does the same on demand, which is how an operator restores balance right after a rolling restart instead of waiting 5 minutes.
- 3 brokers
- payments: 1 partition, replication factor 3
- Broker 2 out of sync: Only broker 1 is eligible once broker 2 is removed
- Both followers in sync: List order picks broker 2
Strike the lever to kill broker 0, the leader. In one variant broker 2 falls behind and leaves the in-sync set, so broker 1 must lead; in the other, list order picks broker 2.
The replica list is 0, 2, 1. With both followers keeping up, broker 2 takes over because it comes first. In the slow variant, strike after the leader has removed broker 2 and the controller skips it for broker 1; strike before the removal and broker 2, still in sync, leads, holding everything readers were given because the mark was pinned to its progress. Broker 0 returns as a follower.
About this lesson
This section is part of the Replication: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.