Ordering Under Retries

Concepts covered: Idempotent producers

5 requests in the air

max.in.flight.requests.per.connection lets the sender have up to 5 produce requests to 1 broker unanswered at once. Throughput depends on it: 1 in flight means 1 request per round trip, 5 means 5. It also opens a door. Batch A with records 10 to 12 for partition 0 goes out at t=0; batch B with records 13 and 14 for the same partition goes out at t=2 while A is still unanswered. The leader rejects A with NotEnoughReplicas because the in-sync set dipped below the minimum for an instant, then the follower rejoins and the leader appends B at t=5. The producer waits out retry.backoff.ms and retries A at t=102; the leader appends it at t=105. The partition now reads 13, 14, 10, 11, 12.

A reader of this partition sees record 13 before record 10. If those are 2 updates to the same account, the older one applies last and wins. Nothing was lost and nothing was duplicated; the log holds every record once, in an order the producer never sent. Order within a partition is the promise Kafka makes, and this scene breaks it from the producer's side.

Fix 1: one request at a time

Setting max.in.flight.requests.per.connection=1 closes the door with no help from the broker: B can't leave until A is answered, so a rejected A is retried before B ever goes. The price is throughput per broker connection, which falls to 1 request per round trip. A producer whose broker answers in 10 ms is capped at 100 requests a second per broker. How many records each of those 100 carries is then up to linger and batch.size. Producers that wanted order with retries before idempotence existed ran this way and paid for it.

Fix 2: let the leader refuse the overtaker

The idempotent producer keeps order at 5 in flight because the leader checks sequences exactly. Say this producer has already sent 10 records to partition 0, so A's first sequence is 10 and B's is 13, matching the record labels. The leader rejects A as before. When B arrives, the leader is expecting sequence 10. It finds 13 and rejects B with OutOfOrderSequenceException instead of appending it. The producer reads that as a signal to retry in sequence order: it re-queues A ahead of B, sends A, waits for A's acknowledgement, then sends B. The log ends with 10 through 14 in order, and at every other moment up to 5 requests stay in flight.

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000

How long a retry keeps trying

delivery.timeout.ms bounds the retrying, with retries left at its default. The 4.4 producer retries until that budget, 120000 ms by default, runs out, measured from the moment send() returned, and the budget covers linger, every attempt's request.timeout.ms of 30000 and every retry.backoff.ms between attempts. A batch still unacknowledged when the budget ends fails with a TimeoutException to the application, and the batches behind it in the same partition continue with their sequences adjusted so the gap does not block them.

Order was only ever promised inside 1 partition. Records with different keys that land on different partitions are appended by different leaders with no clock between them, and a reader of partition 2 may see a record sent at t=9 before a reader of partition 0 sees one sent at t=8. The settings in this section keep the sequence a producer sent to 1 partition; a workload that needs 2 records in order puts them under 1 key.

max.in.flight=1, idempotence off

• 1 request outstanding per broker
• Order kept by never letting a second request leave
• Throughput capped at 1 request per round trip
• No sequence checks on the broker

Idempotent, max.in.flight=5

• Up to 5 requests outstanding per broker
• Order kept by the leader rejecting a batch whose sequence skips ahead
• Throughput of up to 5 requests per round trip
• Needs acks=all and a producer id

About this lesson

This section is part of the Producers: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.