Sending a Record
Concepts covered: Batching and compression
The producer lives in your application
A producer is a library object inside your own program, built from a handful of settings and kept open for the life of the process. On startup it connects to one of the brokers listed in bootstrap.servers, asks it for metadata, and learns from the reply every partition of every topic it writes to and which broker currently leads each one. From then on it talks to leaders directly: a record bound for partition 2 of trip-pings travels to the broker that leads partition 2, with nothing in between. If a leader moves, the next metadata refresh tells the producer where.
# producer.properties
bootstrap.servers=broker-0:9092,broker-1:9092,broker-2:9092
client.id=driver-app-pings
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializerOnly bootstrap.servers and the 2 serializer lines are required. The broker list is a starting point, so any live broker from it is enough to learn the whole cluster. The serializers name how the key and value are turned into bytes, because a broker stores bytes and nothing else; StringSerializer writes the UTF-8 bytes of a string, and a later lesson covers the other formats. client.id is a label the brokers attach to this producer's requests in their logs and metrics, so a busy producer can be found by name.
The command-line producer shipped with Kafka 4.4 wraps the same library: kafka-console-producer.sh --bootstrap-server broker-0:9092 --topic trip-pings reads a line from the terminal and calls send() for each one, so every step below happens for a line you type.
6 steps between send() and the log
| Step | Where it runs | What is settled there |
|---|---|---|
| 1. Serialize | Producer, on the calling thread | The key and value become byte arrays |
| 2. Pick a partition | Producer, on the calling thread | Which partition of the topic the record will live in |
| 3. Park in the buffer | Producer, on the calling thread | send() returns a future; the record waits in memory |
| 4. Ship to the leader | Producer, sender thread | A produce request carries the record to the partition's leader |
| 5. Append | Leader broker | The record gets the next offset and lands at the end of the log |
| 6. Reply | Leader broker, then sender thread | The future completes, or the callback runs, with the offset |
The first 3 steps run on the thread that called send(), and they take microseconds. Nothing has crossed the network when send() returns: the record sits in a memory buffer inside the producer, and the call hands back a future that will hold the result. Step 4 belongs to a second thread the producer started on its own, the sender. It drains the buffer, groups the records headed for each partition, and writes one produce request per leader. The leader appends each record at the end of its partition's log, stamps it with the next offset, and the followers copy it as they do any other write. The reply travels back along the same connection, and the sender completes the future for every record the request carried.
- App thread (source, calls send())
- Producer buffer (queue, KafkaProducer)
- Sender thread (transform, KafkaProducer); parallelism: 1 sender thread
- Leader of partition 2 (storage, broker 1)
- Followers (storage, brokers 0 and 2)
- App thread -> Producer buffer
- Producer buffer -> Sender thread
- Sender thread -> Leader of partition 2
- Leader of partition 2 -> Followers
What the producer gets back
{
"topic": "trip-pings",
"partition": 2,
"offset": 48213,
"timestamp": 1762243200517
}The reply is a RecordMetadata: the topic, the partition the producer chose, and the offset the leader assigned. The partition was known before the request left, because the producer picked it. The offset was known to nobody until the append, because only the leader knows how long its log is at that moment, and 2 producers writing to the same partition get interleaved offsets in whatever order their requests arrived. Your code reaches the metadata in 1 of 2 ways: call get() on the future, which blocks the calling thread until the reply, or pass a callback to send(), which the sender thread invokes when the reply comes. The callback form keeps the application thread free to keep sending while earlier records are still in flight.
| Clock | Where | Event |
|---|---|---|
| 0.00 ms | App thread | send() called with key trip-8841 |
| 0.02 ms | App thread | Partition 2 chosen, record parked, future returned |
| 0.90 ms | Sender thread | Produce request to broker 1, leader of partition 2 |
| 1.40 ms | Broker 1 | Record appended at offset 48213 |
| 2.10 ms | Brokers 0 and 2 | Followers fetch the record |
| 2.60 ms | Broker 1 | Reply sent to the producer |
| 3.10 ms | Sender thread | Future completed, callback runs with offset 48213 |
The 3 ms in this timeline is a round trip inside 1 datacenter; across regions the same 7 lines stretch to tens of milliseconds, and almost all of the added time sits between the sender's request and the leader's reply. The leader answers once the record is as safe as the producer asked it to be, and the section after next sets that bar.
About this lesson
This section is part of the Producers: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.