Replicas Across Racks
Concepts covered: Replication and leaders
Six brokers run across 3 availability zones, 2 brokers in each: brokers 0 and 3 in zone A, 1 and 4 in zone B, 2 and 5 in zone C. A topic with 6 partitions and replication factor 3 is created on a cluster that knows nothing about the zones, so the controller spreads replicas across brokers and nothing else. A partition placed on brokers 0, 3 and 1 has 2 of its 3 copies in zone A, and when zone A loses power it keeps 1 in-sync replica, below a minimum of 2, and refuses writes until the zone returns. Nothing was lost, and the partition is still down.
Telling the broker where it is
# server.properties on broker 3, which runs in zone A
node.id=3
broker.rack=zone-abroker.rack is a label and nothing more; the broker does not check it against anything. The controller reads it when a topic is created or gains partitions and places each partition's replicas so that they spread across as many distinct racks as the replication factor allows, which with 3 racks and replication factor 3 means 1 replica per rack for every partition. It also rotates which broker holds the first replica, so leadership spreads evenly. The placement happens once, at creation: adding a rack label to a running broker changes nothing about partitions that already exist.
| Partition | Replicas (brokers) | Racks | Leader |
|---|---|---|---|
| 0 | 0, 1, 2 | A, B, C | 0 |
| 1 | 1, 2, 3 | B, C, A | 1 |
| 2 | 2, 3, 4 | C, A, B | 2 |
| 3 | 3, 4, 5 | A, B, C | 3 |
| 4 | 4, 5, 0 | B, C, A | 4 |
| 5 | 5, 0, 1 | C, A, B | 5 |
Every partition has 1 copy in each zone, and each broker leads exactly 1 partition and follows 2. With min.insync.replicas set to 2, the table also tells you what the cluster can survive.
- Broker 0 (storage, zone-a)
- Broker 3 (storage, zone-a)
- Broker 1 (storage, zone-b)
- Broker 4 (storage, zone-b)
- Broker 2 (storage, zone-c)
- Broker 5 (storage, zone-c)
- Broker 0 -> Broker 1
- Broker 0 -> Broker 2
- Zone A: Broker 0, Broker 3
- Zone B: Broker 1, Broker 4
- Zone C: Broker 2, Broker 5
Zone B loses power
Brokers 1 and 4 vanish together. The controller fences both, removes them from every in-sync set, and elects new leaders for partitions 1 and 4, which they led, by the rule from the previous section. The next table is the cluster 1 second later.
| Partition | Leader | In-sync set | Fully acknowledged writes with a minimum of 2 |
|---|---|---|---|
| 0 | 0 | 0, 2 | accepted |
| 1 | 2 | 2, 3 | accepted |
| 2 | 2 | 2, 3 | accepted |
| 3 | 3 | 3, 5 | accepted |
| 4 | 5 | 5, 0 | accepted |
| 5 | 5 | 5, 0 | accepted |
Leadership is now uneven: brokers 2 and 5 lead 2 partitions each while brokers 0 and 3 lead 1, until zone B returns and the preferred leader check moves partitions 1 and 4 back. Without rack labels, a partition on brokers 1, 4 and 2 would now sit at 1 in-sync replica and refuse writes, and nobody would have known it was exposed until the zone went dark.
Three numbers that have to agree
Replication factor 3 across 3 racks with a minimum of 2 survives any 1 rack and keeps writing. The same topic on a cluster with only 2 racks puts 2 replicas in one rack for every partition, so losing a rack drops every partition that had 2 copies there to 1 in-sync replica, below the minimum, and roughly half the partitions stop accepting writes even though none lost data. Replication factor 3 across 3 racks with a minimum of 3 keeps 3 copies at all times and stops writing on every partition the moment any rack blinks. Count the racks first, then pick the other 2 numbers.
The price is network. With 1 replica per zone, every record a leader receives crosses 2 zone boundaries to reach its followers, and on a public cloud, traffic between zones is typically billed by the gigabyte while traffic inside a zone usually is not. A topic taking 100 MB per second of writes sends 200 MB per second across zone boundaries for replication alone, before any consumer reads it. That is what surviving a zone loss costs, paid on every byte written.
About this lesson
This section is part of the Replication: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.