Minimum In-Sync Replicas

Concepts covered: In-sync replicas

Replication factor 3 promises 3 copies of every record, but the promise only holds while 3 brokers are up. With min.insync.replicas at its default of 1, a partition whose 2 followers have both dropped out of the in-sync set keeps accepting writes with the leader as the only replica, and the leader confirms each record as fully replicated because, among the in-sync replicas, it is. If that broker's disk fails before the followers return, those records are gone. min.insync.replicas sets the floor: the number of in-sync replicas a partition must have before it accepts a write that asked for full replication.

Which writes it refuses

A producer chooses how much confirmation it wants through its acks setting. With acks=all, the leader confirms a record only after every in-sync replica has it, which in high watermark terms means after the mark has passed the record. That is the write min.insync.replicas guards. When the in-sync set is smaller than the minimum, the leader answers an acks=all send with the error NotEnoughReplicas and appends nothing. A producer sending with acks=1 asked only for the leader's confirmation, and the leader keeps accepting those writes with 1 copy, minimum or not. The setting protects producers that asked for safety and does nothing for ones that did not.

With the eligible leader replicas feature, on by default on clusters newly formatted with a recent 4.x release and switched on by an operator after an upgrade, the rule also reaches the high watermark: while the in-sync set is below the minimum, the leader stops advancing the mark. Records that arrived with acks=1 during the shortage stay unreadable until enough replicas are back in sync, so no reader acts on a record only 1 disk holds.

The error is retriable: the producer's client resends on its own, so a shortage of a few seconds looks to the application like a slow send. One that outlasts the client's patience surfaces as an exception, which tells the application its data is not being stored to the standard it asked for.

Setting it

The setting lives on the topic, with a broker-wide default for topics that do not set their own. The usual pairing is replication factor 3 with a minimum of 2. kafka-configs.sh --bootstrap-server broker1:9092 --alter --entity-type topics --entity-name payments --add-config min.insync.replicas=2 raises it on 1 topic at once. The broker properties below set the floor for every topic without its own value, existing topics included, once each broker restarts with them; default.replication.factor shapes only topics created afterwards.

# server.properties on every broker
default.replication.factor=3
min.insync.replicas=2

The trade behind the number

Raising the minimum trades availability for durability. At replication factor 3 and a minimum of 2, 1 broker can be down and every confirmed record still sits on 2 disks. A minimum of 3 guarantees 3 copies, and the first broker restart stops writes on every partition that broker holds. A minimum above the replication factor can never be met, so the topic refuses every fully replicated write, and the cluster accepts that configuration without complaint.

Refusing a write pushes the problem back to the producer, which retries, buffers, or fails loudly, instead of quietly holding data on 1 disk. For a payments topic the refusal is the right answer: a 30 second pause while a broker comes back costs less than a lost transfer. For a topic of debug metrics sampled every second, a minimum of 1 and the occasional lost record may be the better trade. The number encodes a business decision about which failure you would rather explain.

With 1 broker down the in-sync set holds 2 replicas, the minimum, and both variants keep writing. The second kill drops the set to 1. In the acks=all variant every send from that point comes back with NotEnoughReplicas and the leader's log stops growing; in the acks=1 variant the log keeps growing on a single disk, which is the situation the setting exists to prevent.

Do

• Pair replication factor 3 with min.insync.replicas=2 for most topics
• Set the broker default so every topic without its own value inherits it
• Alert on UnderMinIsrPartitionCount, since producers are being refused
• Raise the minimum to the replication factor only where pausing beats losing a record

Avoid

• Leave min.insync.replicas at 1 on a topic that cannot lose records
• Set the minimum above the replication factor
• Expect it to protect producers that send with acks=1
• Treat NotEnoughReplicas as a producer bug; it is a broker count

About this lesson

This section is part of the Replication: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.