Replication Factor

Concepts covered: Replication and leaders

The number of copies is a choice per topic

Replication factor is the count of replicas a topic keeps for each partition, chosen when the topic is created with --replication-factor, or taken from the broker's default.replication.factor when the flag is left out. A factor of 1 keeps 1 copy, which means the partition has a leader and no followers; the loss of that broker takes the partition offline until the broker returns, with its records gone for good if the disk went with it. kafka-topics.sh --describe prints the factor on its first line, before the per-partition rows.

Failures each extra copy survives

The middle column is the reason to replicate. With factor 3, a partition keeps serving reads and writes through the loss of any 2 of its 3 brokers, because 1 in-sync copy remains to lead. The jump from 1 to 2 buys the most: it turns a broker failure from an outage into a change of leader. The jump from 2 to 3 buys the ability to take 1 broker down on purpose, for an upgrade or a disk swap, while still holding a spare copy in case a second broker fails during the work.

Disk, network and broker time

Each copy costs disk, network and broker time. A topic holding 200 GB of records occupies 600 GB across the cluster at factor 3, because each of 3 brokers writes the full log. Each record a producer sends crosses the wire once to the leader and then once more to every follower, so factor 3 turns 200 GB of produced data into 400 GB of fetch traffic between brokers. And every follower's fetch loop is work on the leader's side too, answering requests for every partition it leads, so a leader of 500 partitions at factor 3 answers 1,000 follower loops.

Latency is the cost people expect. Whether the leader waits for followers to copy a record before confirming it to the producer is a producer setting, taught in the producers lessons, and that setting is where the factor touches write latency: each follower that must copy first adds a round trip inside the cluster before the confirmation leaves. In a healthy cluster that round trip is a millisecond or two; on a cluster with 1 slow follower, it is however slow that follower is, which is 1 more reason the ISR rule from the last section drops a stuck follower rather than waiting on it.

Worked through for 1 topic: 12 partitions that have accumulated 350 GB of records and take in 50 GB a day. At factor 1 that is 350 GB on disk across the cluster; at factor 3, 1,050 GB, spread over 36 replicas, so a 6-broker cluster carries about 175 GB of it per broker. The same topic at factor 3 moves 100 GB a day of follower fetch traffic between brokers, on top of the 50 GB the producers send in.

Factor 3 for most production topics

A factor of 3 keeps serving through 1 planned outage plus 1 surprise. During a rolling restart of a 6-broker cluster, 1 broker is always down on purpose; with factor 2, every partition that broker holds is left with a single copy for the duration, and a second broker's disk failing in that window loses records. With factor 3, the same window leaves 2 copies. Going to 4 buys a third simultaneous failure, which is rarely worth 33% more disk and bandwidth, so most production topics stop at 3. A topic whose records could be regenerated from a source system, such as a nightly export, can run at 2 to save disk, and the team owning it should be able to say so out loud.

# applies to topics created without --replication-factor,
# including topics a client creates by naming one that does not exist
default.replication.factor=3

Changing a topic's factor after creation means adding or removing a replica for every partition, which is a partition move: the new broker must copy the full log from the leader before it is a replica worth having, and for the 350 GB topic above that is 350 GB of fetch traffic to add 1 copy. Kafka 4.4 does it through kafka-reassign-partitions.sh, and a later lesson covers the tool. The practical point now is that the factor is a creation-time decision that costs a full copy of the data to revise, so it is worth getting right on the day the topic is created.

Do

• Set default.replication.factor=3 on every broker so an accidental topic still has copies
• Choose the factor from how the topic's records would be recovered if lost
• Keep factor 3 or more for topics whose records exist nowhere else

Avoid

• Create topics at factor 1 when their records matter
• Plan to raise the factor later as though it were a flag flip
• Size disk per topic without multiplying by the factor

About this lesson

This section is part of the Replication: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.