What a Record Carries

Concepts covered: The append-only log

4 parts the producer sets, 2 the broker adds

A record is more than its payload. The producer fills in a key and a value, plus a timestamp and a list of headers, and the broker adds the partition the record landed in and the offset it got. The table lays out all 6, and the rest of the section walks through why each one is there.

Here is the record from the last section, offset 2 of partition 1, written out as a reader would print it. The key and value are bytes on the wire; the printout shows them decoded as text so they can be read.

{
  "topic": "orders",
  "partition": 1,
  "offset": 2,
  "timestamp": 1762516800000,
  "key": "cart-4",
  "headers": {
    "source": "checkout-service",
    "trace_id": "5f3a9c",
    "content_type": "application/json"
  },
  "value": {
    "cart_id": "cart-4",
    "status": "shipped",
    "carrier": "ground",
    "amount_cents": 4599
  }
}

The key

The key is bytes the producer chooses, most often a string naming the thing the event is about: a cart id, a customer id, a device id. The producer client hashes the key to pick a partition before the record leaves it. Every record with key cart-4 hashes to the same number, so every record about cart 4 lands in partition 1, one after another. That is how the order of a partition becomes the order of a cart: placed, then paid, then shipped, received in that sequence by anyone reading partition 1.

A record can be sent with no key. The producer then spreads records across partitions for balance, and 2 records about the same cart may land in different partitions with no order between them. A key is the producer saying which records belong together, and the cost of leaving it empty is paid by every reader that needs them in sequence.

The value

The value is the event, as bytes. Kafka never parses the bytes it stores and returns, so the producer and its readers have to agree on what they mean. They could be text or a JSON document, a number or an image. The orders producer sends a JSON object holding a cart id and a status, along with the carrier and the amount. A reader expecting that shape gets it; a reader expecting a different shape gets bytes it cannot use, and the broker raises no error, because the broker never looked. The agreement is between the teams, and a later lesson covers the tooling that writes it down.

The timestamp

Each record carries a timestamp in milliseconds since 1 January 1970. The one in the printout, 1762516800000, is noon UTC on 7 November 2025. By default the producer stamps the record when it creates it, so the timestamp says when the event happened as the producer saw it. A reader can hand the broker a time and ask for the first offset at or after it, and start reading there. The offset says where a record sits in its partition; the timestamp says when it was made. 2 records can share a timestamp and never an offset.

Headers

Headers are a list of name-value pairs carried alongside the value. Each name is a string and each header value is bytes. They hold facts about the record that aren't the event: which service produced it, the trace id that links it to a request, the format of the value, a version of the agreed shape. A reader that filters by those facts reads the headers and leaves the value unparsed, and a tool in the middle can add a header without understanding the payload. The same name may appear more than once in the list.

Do

• Key each record by the thing whose order matters: one cart, one customer, one device
• Put the event itself in the value and keep transport facts out of it
• Keep headers small: a source name, a trace id, a content type

Avoid

• Key by a field that changes during the story, such as order status
• Bury routing facts inside the value, where a filter must parse it to find them
• Send one form of key from one service and another form from a second for the same entity

About this lesson

This section is part of the The Log: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.