Brokers Hold Replicas

Concepts covered: Replication and leaders

A partition is a file, and a file sits on a machine

The log lesson left a partition as an ordered list of records, each at an offset, growing at 1 end. That list is stored on disk, and the disk belongs to a broker: 1 Kafka server process on 1 machine, with an integer id such as 1 or 2. A cluster is a set of brokers, usually 3 or more, each with its own disk and its own network link. Every partition of every topic is stored on at least 1 broker, and the question this lesson answers is what happens to the partition when that broker's disk fails on a Tuesday afternoon.

Replication is the answer. Instead of 1 copy of a partition's log, the cluster keeps several, each on a different broker, and each copy is called a replica. A replica holds the same records at the same offsets as every other replica of its partition, so offset 41 of partition 2 names 1 record on every broker that stores a copy. The number of replicas a topic keeps for each of its partitions is its replication factor, chosen when the topic is created.

Creating a topic with 3 partitions and 3 replicas of each: kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 3 --bootstrap-server broker1:9092. The flag counts every copy, including the one producers will write to, so 3 means 3 brokers each store the full log of each partition.

Suppose orders-2 has received 5 records, at offsets 0 through 4. Broker 3's replica has those 5 records at offsets 0 through 4, broker 4's replica has the same 5, and broker 1's replica has the same 5. A consumer that stopped at offset 3 of orders-2 finds the same record at offset 4 whichever copy answers, because offsets belong to the partition and every replica agrees on them. Replication adds copies of the log without adding anything to the log.

Where the copies land

The cluster assigns replicas when the topic is created, and no 2 replicas of 1 partition ever share a broker. With 4 brokers and replication factor 3, each partition lands on 3 of the 4: the first replica shifts 1 broker along with each new partition, and the other 2 go to the brokers after it in id order, wrapping from 4 back to 1. Topics with more partitions than brokers go round again, shifting which brokers follow on each pass.

Read the table by column and the point of the rotation shows: broker 1 stores replicas of 3 partitions, and so does each of the other 3 brokers, so disk use is even. Read it by row and no partition depends on 1 machine. Unplug broker 3 and orders-0 still lives on brokers 1 and 2, orders-1 on brokers 2 and 4, orders-2 on brokers 1 and 4, and orders-3 never touched broker 3 at all. Every record of every partition is still on a disk somewhere in the cluster.

kafka-topics.sh --describe --topic orders lists, for each partition, the ids of the brokers holding a replica, in assignment order, under the heading Replicas. That column is the first thing to read when a broker goes down: every partition whose Replicas list names the dead broker has lost 1 copy, and every partition whose list does not is untouched.

The placement rule sets 2 limits. The replication factor cannot exceed the number of brokers, because the extra copy would have nowhere to go; asking for 4 replicas on a 3-broker cluster fails at creation with InvalidReplicationFactorException and no topic is made. And the factor is per topic, so 1 cluster can keep 3 copies of orders and 1 copy of a scratch topic nobody would miss. Not every copy can take writes from producers, or 3 copies would hand out 3 different offsets for the same record; the next section shows what the other copies do instead.

About this lesson

This section is part of the Replication: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.