Idempotent Producers
Concepts covered: Idempotent producers
One send, 2 records
A retry is only safe when the first attempt did not land. The case that breaks that assumption is a lost answer. A producer sends a batch with acks=all; the leader appends it, then waits for the followers to copy it before answering. One follower is slow, the answer misses request.timeout.ms, and the producer, which cannot tell a slow answer from a lost request, sends the batch again. The leader has no memory of the first attempt beyond the bytes in its log, so it appends the batch a second time. The partition now holds the same payment twice, and every reader will see both.
A producer id and a sequence number
With enable.idempotence=true, the 4.4 default, the producer asks a broker for a producer id on its first send and then stamps every batch with that id and a sequence number. The sequence counts records per partition from 0, so a batch carries its first record's sequence and the leader derives the rest from the record count. The leader keeps, for each producer id on each partition, the sequence ranges of the last 5 batches it appended. A batch whose sequence it already appended is answered with the original offset and appended again never. A batch whose first sequence is higher than the next expected number is rejected with OutOfOrderSequenceException, because something before it went missing.
| attempt | batch carries | leader expects | leader does | log after |
|---|---|---|---|---|
| 1 | pid 7, seq 0, 3 records | seq 0 | appends at offsets 40 to 42, waits for followers | 40, 41, 42 |
| answer lost | producer times out, schedules a retry | 40, 41, 42 | ||
| 2, the retry | pid 7, seq 0, 3 records | seq 3 | finds seq 0 in its window, answers offset 40 | 40, 41, 42 |
| 3, next batch | pid 7, seq 3, 2 records | seq 3 | appends at offsets 43 and 44 | 40 to 44 |
- 3 brokers
- payments: 1 partition, replication factor 3
- enable.idempotence=true: The same lost answers with a producer id and sequence on the batch
A slow follower holds the acks=all answer past request.timeout.ms, so the producer retries. Without idempotence each retry is appended; with it, the leader answers the duplicate from its window.
In the scene every attempt reaches the same leader, so the difference lies in what the leader remembers. Without idempotence the leader sees 3 arrivals of the same bytes and appends 3 times; the followers copy all 3 and the partition holds 3 charges of 40.00 for 1 invoice. With idempotence the second and third arrivals carry sequence 0 again, the leader finds sequence 0 already in its window, answers with the offset it wrote the first time, and appends nothing.
Idempotence needs acks=all and a positive retries. It also caps max.in.flight.requests.per.connection at 5, which matches the window the leader keeps. The 4.4 producer applies these for you when the setting is on, and refuses to start with a ConfigException if you set one of them to a conflicting value, such as acks=1.
The window of 5 is also why max.in.flight.requests.per.connection is capped at 5 for an idempotent producer. The leader recognises a duplicate only of a batch it still remembers, and with 6 requests in flight the oldest could fall out of the window before its retry arrived, turning a safe retry into a second copy.
What the window does not cover
Duplicates idempotence does not stop
- Two calls to send() with the same payload: the producer assigns 2 sequences and the leader appends both.
- A producer process that crashes and restarts: the new process gets a new producer id, so a re-sent record carries a sequence the leader has never seen under that id.
- Two producers sending the same record: each has its own producer id, and the leader deduplicates within 1 id only.
The producer id and sequence travel in the batch header, so they sit in the log on every replica. A follower that becomes leader rebuilds the per-producer state from its own copy of the log, and a retry that arrives at the new leader is deduplicated the same way. The window survives the failure, which matters because a leader change is exactly when retries happen.
About this lesson
This section is part of the Producers: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.