When a Broker Dies
Concepts covered: Replication and leaders
Broker 2 and its 3 replicas
Broker 2 on the 4-broker cluster loses power at 14:02:10. It held 3 replicas: the leader of orders-1 and followers of orders-0 and orders-3. The controller, the part of the cluster that tracks which brokers are alive, stops receiving broker 2's heartbeats; after broker.session.timeout.ms, 9000 milliseconds by default, it declares broker 2 gone, removes it from the ISR of every partition it held, and for each partition it led picks a new leader from the remaining ISR. Then 2 different things happen, depending on the role broker 2 played in each partition.
For orders-0 and orders-3, where broker 2 was a follower, the ISR shrinks from 3 ids to 2 and nothing else changes: the leaders keep appending, the other follower keeps fetching, and producers and consumers of those partitions see nothing at all. For orders-1, where broker 2 led, every producer and consumer has lost its point of contact, and the partition is without a leader until the controller names one.
Picking a new leader
The ISR of orders-1 was brokers 2, 3 and 4 until the crash. The controller removes 2 and chooses the first surviving id in Replicas order, broker 3. It records the change and tells every broker; from that moment broker 3 appends writes for orders-1 and broker 4 fetches from broker 3. Because broker 3 was in sync, it holds the records the dead leader had copied out, and new offsets continue from the end of broker 3's log. A record that broker 2 took in and no follower had fetched yet is the one at risk, and whether the producer was ever told that record was safe depends on the producer setting the producers lessons teach.
| Time | Event | A producer writing to orders-1 sees |
|---|---|---|
| 14:02:10 | broker 2 loses power | its open send gets no reply |
| 14:02:19 | controller declares broker 2 gone, broker 3 leads orders-1 | still waiting on a silent connection |
| 14:02:40 | the producer's request timeout expires | metadata refresh names broker 3 |
| 14:02:40.1 | broker 4 already fetches from broker 3 | the resend succeeds at the next offset |
The pause clients see
The pause has 2 parts. The controller needs around 9 seconds, the session timeout, to elect a new leader, and a client uses that leader only after it gives up on broker 2. When the broker process crashes on a running machine, the operating system resets the connection, requests fail at once, and the client retries until the election lands: about 10 seconds with defaults. A machine that loses power sends nothing back, so the client waits out its request.timeout.ms, 30000 by default, then refreshes metadata and goes to broker 3: about 30 seconds. Consumers do the same with their fetches, resuming from the offset they had reached.
- 3 brokers
- payments: 1 partition, replication factor 3
- No failure: Broker 2 leads throughout; the followers fetch and the consumer reads from broker 2.
- Broker 2 killed at 2 s: The controller declares broker 2 gone, broker 3 becomes leader, the producer and consumer retry against broker 3, and broker 2 returns at 6 s as a follower.
Broker 2 leads payments-0, with brokers 3 and 1 following. In the second variant broker 2 is killed at 2 s and returns at 6 s; the session timeout is 1 s here so the election fits.
A dead leader's partitions each pass to the next in-sync broker in their Replicas list. On the 4-broker layout broker 2 leads only orders-1, so broker 3 inherits it alone. On topics with many more partitions than brokers, each pass of the assignment puts a different broker second, so the leadership of a broker that led 100 partitions, and its producers' retries, splits across the survivors.
Broker 2 returns and takes orders-1 back
At 14:40 broker 2 is powered on. Its 3 replicas fetch what they missed, including 38 minutes of orders-1 records from broker 3, and re-enter their ISRs. Broker 2 is still first in the Replicas list of orders-1, and with auto.leader.rebalance.enable on by default, the controller checks every 300 seconds for partitions not led by their preferred leader, so leadership moves back to broker 2 within 5 minutes of catching up. Broker 3 is up for that move and answers clients with NOT_LEADER_OR_FOLLOWER at once, so they redirect after 1 metadata refresh.
A second failure before the first is repaired is where factor 3 and the ISR meet. If broker 3 also dies before broker 2 returns, orders-1 has 1 replica left, broker 4, which is in the ISR and becomes leader; the partition keeps serving with no spare. If broker 4 then fails too, no in-sync replica remains and the partition goes offline: sends fail and fetches return nothing until a broker holding a copy comes back. With factor 3 that takes 3 failures of exactly the right brokers before anyone repairs the first, which is what the factor was chosen to make rare.
About this lesson
This section is part of the Replication: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.