The Durability Contract
Concepts covered: Producer acknowledgements
acks names who must hold the record
acks sets how many copies exist when the producer counts a send as done. With acks=0 the producer counts it done when the bytes leave the socket, so a leader that dies while the request is in the air loses the record and the producer never hears. With acks=1 the leader answers after appending to its own log, before any follower has fetched the batch; if the leader dies in that gap, the new leader is a follower that never got the record, and the record is gone after the producer was told it was safe. With acks=all, the 4.4 default, the leader answers only when every replica in the in-sync set has appended the batch, which is the moment the high watermark passes it.
| acks | leader answers when | lost if | wait per request |
|---|---|---|---|
| 0 | nothing is awaited | the socket drops or the leader dies with it in flight | none |
| 1 | its own append completes | the leader dies before a follower fetches | 1 round trip |
| all | every in-sync replica has appended | every in-sync replica dies | 1 round trip plus the slowest in-sync follower's fetch |
acks=0 has a place: a stream of sensor samples where the next reading replaces the last, sent by a producer that must never wait on the cluster. Everywhere a record is a fact rather than a sample, it's the wrong setting.
min.insync.replicas puts a floor under all
acks=all waits for the in-sync set, and the in-sync set shrinks when followers fall behind. A topic with 3 replicas whose 2 followers both fell out of sync has an in-sync set of 1, and acks=all then means the leader alone, the same promise as acks=1. min.insync.replicas on the topic sets the smallest in-sync set the leader will accept an acks=all write with. At 2, a leader whose in-sync set has dropped to itself rejects the batch with NotEnoughReplicas. That error is retriable, so the producer retries until a follower catches up or delivery.timeout.ms expires. The count includes the leader, so 2 means the leader plus 1 follower.
A sibling error, NotEnoughReplicasAfterAppend, arrives when the in-sync set shrank after the leader appended the batch. The record sits in the leader's log unacknowledged, and the retry that follows is exactly the retry the previous sections made safe: the idempotent leader answers it from the window without a second append.
| replicas | min.insync.replicas | acks | replica brokers down before writes stop | can an acked record vanish when 1 broker dies? |
|---|---|---|---|---|
| 3 | 1 | 1 | 2 | yes, if the leader dies before a follower fetches |
| 3 | 1 | all | 2 | yes, once the in-sync set has shrunk to the leader |
| 3 | 2 | all | 1 | no |
| 3 | 3 | all | 0 | no, and any single failure stops writes |
| 2 | 2 | all | 0 | no, and any single failure stops writes |
The row to run is 3 replicas, min.insync.replicas=2, acks=all. Any 1 broker can die and the partition keeps taking writes, because 2 in-sync copies remain; and every acknowledged record sits on at least 2 brokers, so the death of 1 can't remove it. Lose 2 of the 3 and writes stop rather than proceed on a single copy, which is the point: the producer gets an error it can retry instead of an acknowledgement it can't trust.
- 3 brokers
- ledger: 1 partition, replication factor 3
- acks=all: The same sends acknowledged only once every in-sync replica has the batch
Records arrive every 60 ms while followers copy with a 120 ms delay. Strike the leader: under acks=1 records already acknowledged can be on no surviving broker; under acks=all every acknowledged record already sits on the followers.
Set the floor on the topic so each topic states its own contract: bin/kafka-configs.sh --bootstrap-server broker0:9092 --alter --entity-type topics --entity-name payments --add-config min.insync.replicas=2. A topic created without it inherits the broker's min.insync.replicas, which defaults to 1.
The contract has 3 parties
The producer's acks, the topic's min.insync.replicas and the topic's replication factor are 3 settings owned by 3 people, and the promise holds only when all 3 agree. A producer at acks=all writing to a topic with min.insync.replicas=1 has a contract of 1 copy the moment a follower lags. A topic at min.insync.replicas=2 written by a producer at acks=1 never consults the floor, because the leader checks the floor only for acks=all requests. A topic with 2 replicas and min.insync.replicas=2 holds the promise but stops writing the moment either broker restarts for an upgrade.
Readers close the loop. A consumer fetches up to the high watermark, and the high watermark advances only when every in-sync replica has the record, so a record acknowledged under acks=all with a floor of 2 is readable the instant the producer hears back and sits on 2 brokers when it's read. A record acknowledged under acks=1 becomes readable only when the followers catch up, and if the leader dies first it never does.
- Do
- Set acks=all and min.insync.replicas=2 on every topic whose records you cannot replay
- Keep the replication factor 1 above min.insync.replicas so 1 restart doesn't stop writes
- Treat NotEnoughReplicas alerts as the floor doing its job, and page on them
- Avoid
- Rely on acks=all alone while min.insync.replicas sits at 1
- Set min.insync.replicas equal to the replication factor on a topic that must stay writable
- Read acks=1 throughput figures as the cost of acks=all
About this lesson
This section is part of the Producers: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.