Unclean Leader Election

Concepts covered: In-sync replicas

A clean election picks the new leader from the in-sync replicas, so the new leader holds every record the old one had committed. That rule has a gap. The ISR can shrink to the leader alone when every follower falls behind, and if that leader then dies, no replica is in sync with it. The controller has 2 options: leave the partition without a leader until the dead broker returns, or make a replica that is missing records the leader. The setting that picks between them is unclean.leader.election.enable.

A partition with one replica left in sync

Take partition 3 of orders, replicated on brokers 0, 1 and 2 with broker 0 leading. Both followers sit on a rack whose switch starts dropping packets, their fetches stall, and after replica.lag.time.max.ms the leader removes them from the ISR. Writes keep landing: min.insync.replicas is 1, the ISR is {0}, and the high watermark advances with every record. The log end offsets drift apart.

Broker	Role	Log end offset	In the ISR
0	leader	100	yes
1	follower	61	no
2	follower	58	no

At offset 100, broker 0 loses power and goes down. Its disk still holds offsets 0 to 99, but no one can read them until the machine boots again. The controller sees a dead leader and an ISR with nobody else in it.

Off: wait for the leader

With the setting at its default of false, the controller leaves partition 3 with no leader. Writes to it fail, readers on it get nothing, and the other partitions of orders carry on. When broker 0 comes back with its log intact, it resumes as leader at offset 100, the followers catch up, and every acknowledged record is still there. The price is that this partition stays offline for as long as broker 0 is down.

On: promote a stale replica

With the setting at true, the controller makes a replica from outside the ISR the leader. It never sees the replicas' log end offsets, so it does not look for the freshest one. It takes the first live broker in the partition's replica list, here [0, 1, 2], which is broker 1. Broker 1 leads from its own log end, offset 61, and the ISR becomes {1}. Records 61 through 99, all acknowledged, now exist only on broker 0's powered-off disk. Had the list been [0, 2, 1], broker 2 would lead from offset 58 and records 58 to 99 would be lost, 3 more than here, while broker 1 sat holding them. New writes arrive at once and the leader assigns them offsets starting at 61. Offset 80 now names a record written after the failure, and any reader that already consumed the old record 80 holds a value the cluster no longer has.

The last in-sync replica dies

3 brokers
orders: 1 partition, replication factor 3

Election off (default): No in-sync replica remains, so the partition has no leader and the sends after the kill fail. Broker 0 then returns with offsets 0 to 6 intact, leads again, and the next send lands at offset 7.
Election on: The first live broker in the replica list, broker 1, becomes leader at its own log end; records 4 to 6 are gone and the new sends take their offsets.

Both followers stop fetching and leave the ISR, so records 4 to 6 reach broker 0 alone. When broker 0 dies, the switch decides whether the partition waits for broker 0 to return or lets a stalled follower lead from its shorter log.

The old leader truncates on return

Broker 0 restarts holding offsets 0 to 99 and finds broker 1 leading with a newer leader epoch. It asks the leader where their logs diverge, learns that the leader's epoch began at offset 61, truncates its own log to offset 61, and fetches the new records from there. The 39 records it had are gone from the cluster for good. A reader whose position sat inside the lost range meets one of 2 outcomes, depending on where that position falls against the new leader's log end. A reader at offset 90 while the log ends at 65 asks for an offset that does not exist, gets an offset-out-of-range error, and must restart from the log's start or its end by whatever rule it was configured with. A reader at offset 63 gets no error at all: it reads on, and records 63 to 65 are ones written after the failure, different from the records it would have read there before. Nothing warns a reader that offsets 61 to 99 now hold different records; the only trace is a jump in the leader epoch, which the broker writes to its own log file.

Moment	Broker 0 log	Broker 1 log	Readable range
before the crash	0 to 99	0 to 60	0 to 99
after the unclean election	offline	0 to 60	0 to 60
after 5 new writes	offline	0 to 65	0 to 65
broker 0 rejoins	0 to 65, truncated	0 to 65	0 to 65

Election off (default)

The partition stays offline until a replica with the full log returns
Every acknowledged record survives
Writers and readers of that partition wait

Election on

A new leader is chosen within seconds
Records only the dead leader held are dropped
Offsets past the divergence point are reassigned to new records

Choosing per topic

The setting is a topic config and overrides the broker default. A topic of page-view counts can run with it on, since a gap of 40 records costs a dashboard a blip. A topic of payments runs with it off and accepts the outage. You can also leave it off everywhere and trigger a single unclean election by hand when the wait has gone on too long: kafka-leader-election.sh --bootstrap-server b1:9092 --election-type UNCLEAN --topic orders --partition 3. That choice is yours at the moment, rather than the controller's by rule.

unclean.leader.election.enable=true

About this lesson

This section is part of the Replication: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.