Replication: Beginner

Creating a topic with 3 partitions and 3 replicas of each: kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 3 --bootstrap-server broker1:9092. The flag counts every copy, including the one producers will write to, so 3 means 3 brokers each store the full log of each partition.

kafka-topics.sh --describe --topic orders lists, for each partition, the ids of the brokers holding a replica, in assignment order, under the heading Replicas. That column is the first thing to read when a broker goes down: every partition whose Replicas list names the dead broker has lost 1 copy, and every partition whose list does not is untouched.

The placement rule sets 2 limits. The replication factor cannot exceed the number of brokers, because the extra copy would have nowhere to go; asking for 4 replicas on a 3-broker cluster fails at creation with InvalidReplicationFactorException and no topic is made. And the factor is per topic, so 1 cluster can keep 3 copies of orders and 1 copy of a scratch topic nobody would miss. Not every copy can take writes from producers, or 3 copies would hand out 3 different offsets for the same record; the next section shows what the other copies do instead.

Take 1 record through the loop. A producer writes order-9001 to orders-0 at 10:00:00.000. Broker 1, the leader, appends it at offset 1204. Broker 2's fetch loop asks for offset 1204 a few milliseconds later and gets it; broker 3's loop does the same. By 10:00:00.020 all 3 replicas hold order-9001 at offset 1204, and the followers are asking for 1205, which does not exist yet, so the leader holds their requests open until a producer sends the next record.

This is the per-partition part of what kafka-topics.sh --describe prints. The Isr column is the subject of the next section; for now read Leader against Replicas and note that each leader is the first id in its own list. Preferred matters later: when a broker returns from a failure, the cluster moves leadership back to the preferred replica so the spread stays even, and section 4 shows that return.

A follower that copies continuously is never more than a few milliseconds behind, but a follower on a sick disk, or behind a saturated network link, can fall far behind, and the cluster needs a crisp rule for which followers still count. The rule defines the in-sync replica set, written ISR. A follower is in the ISR while it has caught up to the leader's end of log at some point in the last replica.lag.time.max.ms milliseconds, 30000 by default. The leader of a partition is always in its own ISR.

Running --describe at 09:00:35 prints Isr: 1,2 against Replicas: 1,2,3: 1 replica is assigned and not caught up. The lag rule covers a follower that is alive and stuck; a broker that stops altogether leaves every ISR sooner, about 9 seconds after it falls silent, through the session timeout section 4 introduces.

Replication factor is the count of replicas a topic keeps for each partition, chosen when the topic is created with --replication-factor, or taken from the broker's default.replication.factor when the flag is left out. A factor of 1 keeps 1 copy, which means the partition has a leader and no followers; the loss of that broker takes the partition offline until the broker returns, with its records gone for good if the disk went with it. kafka-topics.sh --describe prints the factor on its first line, before the per-partition rows.

Changing a topic's factor after creation means adding or removing a replica for every partition, which is a partition move: the new broker must copy the full log from the leader before it is a replica worth having, and for the 350 GB topic above that is 350 GB of fetch traffic to add 1 copy. Kafka 4.4 does it through kafka-reassign-partitions.sh, and a later lesson covers the tool. The practical point now is that the factor is a creation-time decision that costs a full copy of the data to revise, so it is worth getting right on the day the topic is created.

Broker 2 on the 4-broker cluster loses power at 14:02:10. It held 3 replicas: the leader of orders-1 and followers of orders-0 and orders-3. The controller, the part of the cluster that tracks which brokers are alive, stops receiving broker 2's heartbeats; after broker.session.timeout.ms, 9000 milliseconds by default, it declares broker 2 gone, removes it from the ISR of every partition it held, and for each partition it led picks a new leader from the remaining ISR. Then 2 different things happen, depending on the role broker 2 played in each partition.

The pause has 2 parts. The controller needs around 9 seconds, the session timeout, to elect a new leader, and a client uses that leader only after it gives up on broker 2. When the broker process crashes on a running machine, the operating system resets the connection, requests fail at once, and the client retries until the election lands: about 10 seconds with defaults. A machine that loses power sends nothing back, so the client waits out its request.timeout.ms, 30000 by default, then refreshes metadata and goes to broker 3: about 30 seconds. Consumers do the same with their fetches, resuming from the offset they had reached.

At 14:40 broker 2 is powered on. Its 3 replicas fetch what they missed, including 38 minutes of orders-1 records from broker 3, and re-enter their ISRs. Broker 2 is still first in the Replicas list of orders-1, and with auto.leader.rebalance.enable on by default, the controller checks every 300 seconds for partitions not led by their preferred leader, so leadership moves back to broker 2 within 5 minutes of catching up. Broker 3 is up for that move and answers clients with NOT_LEADER_OR_FOLLOWER at once, so they redirect after 1 metadata refresh.