Leaders and Followers
Concepts covered: Replication and leaders
One copy takes the writes
Among a partition's replicas, 1 is the leader and the rest are followers. A producer sends every record for that partition to the leader, and the leader appends it to its log and assigns the offset. Followers write nothing on their own; each asks the leader for the records it does not yet have and appends them to its own log, at the offsets the leader assigned. By default a consumer reads from the leader too, so the leader is the single point of contact for a partition, and the followers are copies kept warm for the day the leader's broker fails.
The reason for exactly 1 writer is the offset. Offsets form a sequence with no gaps, decided once. If broker 1 and broker 2 both accepted records for orders-0, each would hand out offset 17 to a different record, and the 2 logs would disagree about what offset 17 means. A single leader decides the order once, every follower copies that order, and the guarantee from the log lesson holds on every replica: offset 17 of orders-0 is 1 record, everywhere.
- producer (source, client)
- broker 1 (leader) (storage, orders-0 leader)
- broker 2 (follower) (storage, orders-0 follower)
- broker 3 (follower) (storage, orders-0 follower)
- consumer (consumer, client)
- producer -> broker 1 (leader)
- broker 1 (leader) -> broker 2 (follower)
- broker 1 (leader) -> broker 3 (follower)
- broker 1 (leader) -> consumer
The fetch loop
A follower copies the way a consumer reads. It sends the leader a fetch request carrying the next offset it needs; the leader replies with the records from that offset to its log's end; the follower appends them and asks again at once. Each broker runs this fetch loop for every partition it follows, so a follower missing 1 record at 9:00:00.000 usually has it by 9:00:00.050.
| Tick | Leader log end | Follower asks from | Leader returns | Follower log end after |
|---|---|---|---|---|
| 1 | 5 | offset 3 | offsets 3 and 4 | 5 |
| 2 | 8 | offset 5 | offsets 5, 6 and 7 | 8 |
| 3 | 8 | offset 8 | nothing yet | 8 |
| 4 | 9 | offset 8 | offset 8 | 9 |
Take 1 record through the loop. A producer writes order-9001 to orders-0 at 10:00:00.000. Broker 1, the leader, appends it at offset 1204. Broker 2's fetch loop asks for offset 1204 a few milliseconds later and gets it; broker 3's loop does the same. By 10:00:00.020 all 3 replicas hold order-9001 at offset 1204, and the followers are asking for 1205, which does not exist yet, so the leader holds their requests open until a producer sends the next record.
Every broker leads something
Leadership is per partition, and the assignment spreads it: the first broker in a partition's Replicas list is its preferred leader, and because the starting broker rotates from one partition to the next, each broker is the preferred leader of its share. On the 4-broker cluster from the last section, broker 1 leads orders-0, broker 2 leads orders-1, broker 3 leads orders-2 and broker 4 leads orders-3. A broker that leads 1 partition and follows 2 others carries producer and consumer traffic for the first and fetch traffic for the others, which keeps writes from piling up on 1 machine.
| Partition | Leader | Replicas | Isr |
|---|---|---|---|
| orders-0 | 1 | 1,2,3 | 1,2,3 |
| orders-1 | 2 | 2,3,4 | 2,3,4 |
| orders-2 | 3 | 3,4,1 | 3,4,1 |
| orders-3 | 4 | 4,1,2 | 4,1,2 |
This is the per-partition part of what kafka-topics.sh --describe prints. The Isr column is the subject of the next section; for now read Leader against Replicas and note that each leader is the first id in its own list. Preferred matters later: when a broker returns from a failure, the cluster moves leadership back to the preferred replica so the spread stays even, and section 4 shows that return.
Sending to the wrong broker
A producer that sends a record for orders-0 to broker 2 gets an error, NOT_LEADER_OR_FOLLOWER, and broker 2 stores nothing. The producer then asks the cluster for fresh metadata, learns that broker 1 leads orders-0, and resends there. The same error reaches a consumer that fetches from the wrong broker. Leadership can move because of this: nobody edits a client's config, and the client learns the new leader from the next error. Reading from a follower is possible in 4.4 through a consumer setting that a later lesson covers; the default, and the picture to hold now, is leader for both writes and reads.
- Leader
- Receives every record a producer sends for the partition
- Assigns each record its offset
- Answers fetch requests from followers and from consumers
- Exactly 1 per partition at any moment
- Follower
- Fetches from the leader in a loop, like a consumer
- Appends at the offsets the leader assigned
- Accepts no writes from producers
- Ready to become leader, which section 4 shows
About this lesson
This section is part of the Replication: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.