Send Errors and Retries

Concepts covered: Producer acknowledgements

Failures that pass and failures that don't

A produce request can fail for reasons that will pass and reasons that will not. The leader of partition 2 died and a new one is being elected: wait a moment and the same request to the new leader succeeds. The connection to broker 1 dropped: reconnect and try again. The request got no reply within request.timeout.ms: send it again. The producer classifies these as retriable and handles them itself, inside the sender thread, without waking the application. A record larger than the broker accepts, a key that cannot be serialized, a topic name with illegal characters, or a client that lacks permission to write to the topic will fail the same way on the hundredth attempt, so the producer stops at once and hands the error to your code.

The clock that ends a retry

Retries are bounded by time rather than count. delivery.timeout.ms starts when send() is called and, when it runs out, the record fails to the application with a TimeoutException no matter how many attempts were made; the default gives the producer 2 minutes. Inside that window, request.timeout.ms governs 1 attempt: 30 seconds with no reply counts as a failed attempt. Between attempts the producer waits a backoff that starts small and doubles each time up to a ceiling, so a 400 ms leader election costs a few backoffs and metadata refreshes, then 1 retry that lands. The retries setting still exists and still caps the count, but its default is so large that in practice the delivery timeout is what ends the wait.

• 2,147,483,647
• 100 ms
• 1,000 ms

# defaults, written out
delivery.timeout.ms=120000
request.timeout.ms=30000
retry.backoff.ms=100
retry.backoff.max.ms=1000
retries=2147483647

A leader dies mid-send

The application wrote send() once and received 1 callback with an offset, 338 ms later than usual. Had the election outlasted the delivery timeout, that single callback would instead carry a TimeoutException, and your code would decide what to do with pay-7. The leader change itself is the replication lesson's story, playing out exactly as described there; the producer's part is to keep the record in its buffer, refresh metadata, and aim the next attempt at whoever leads now.

A retry can reach a leader that did append the previous attempt before the reply was lost. Kafka 4.4's default producer settings let the broker recognize such a resend and keep 1 copy; the intermediate producer lesson explains how.

The animation sends 8 inventory updates through a partition whose only replica goes down for 1.5 seconds partway through; the single replica keeps anything else from taking over, so the gap stands in for a leader election of that length. Under the default delivery timeout the producer holds the records caught in the gap and lands every one when the broker returns. Switch to a 1 second delivery timeout and the same outage turns those records into errors delivered to the application.

def on_delivery(metadata, error):
    if error is None:
        print(f"landed in partition {metadata.partition} at offset {metadata.offset}")
    elif error.retriable:
        print(f"gave up after the delivery timeout: {error}")
    else:
        print(f"rejected on the first attempt: {error}")

The callback hears every outcome after the record entered the buffer, the same one get() on the future returns; a serializer error never reaches it, because send() throws it. The first branch carries the metadata from section 0. The second fires for a retriable error that outlasted the delivery timeout: decide whether the record is still worth sending and, if so, call send() again. The third fires for an error the producer refused to retry, and resending without changing the record or the configuration fails the same way.

Do

• Keep delivery.timeout.ms longer than the longest leader election you expect
• Log the exception class from the callback, since it tells you which family failed
• Treat a non-retriable error as a bug in the record or the config

Avoid

• Set retries=0 to avoid resends; a leader election then fails sends that would have landed
• Shorten request.timeout.ms below the broker's slowest normal reply
• Swallow callback errors; a send that fails there is otherwise invisible

About this lesson

This section is part of the Producers: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.