Large Messages

Concepts covered: Log storage and retention

Kafka's defaults assume a record under 1 MB. A bigger one is rejected by 2 limits, the producer's and the broker's or topic's, while 2 consumer limits only size fetches. They measure the record as serialized: key, value and headers together, plus the few dozen bytes of batch framing the broker wraps around it. A 3 MB image or a 2 MB JSON document with a base64 blob inside trips the first of them before leaving the producer, and raising that one only moves the failure to the next.

Where a 3 MB record dies

Follow a 3 MB record. The producer serializes it, compares 3145728 bytes with max.request.size, and raises RecordTooLargeException without sending anything. Raise max.request.size to 4 MiB and the producer sends it; the broker compares the batch with the topic's max.message.bytes if one is set, else its own message.max.bytes, and answers with the error MESSAGE_TOO_LARGE, which the producer surfaces as the same exception. Set max.message.bytes=4194304 on the topic as well and the record lands in the log. The consumer then needs nowhere near 3 MB of configuration to read it: since a fetch always returns at least the first batch of the first partition that has data, however large, the record arrives in a fetch of its own. What the consumer does need is memory for it, and a handler written for a payload 30 times its usual size.

The ceiling behind all of these is the broker's socket.request.max.bytes, 104857600, 100 MiB, the largest request of any kind it reads from a connection. A record that needs the limits raised toward that is a sign the record is the wrong unit, which is what the 2 patterns below are for.

Pattern 1: the claim check

Store the payload somewhere built for payloads and send a record that points to it. The producer writes the 3 MB image to an object store under a key it chooses, then sends a small record carrying that key, the size, a checksum and the content type. The consumer reads the record, fetches the image by the key, verifies the checksum, and processes it. The log holds 300 bytes per event, every limit stays at its default, and readers that only need the metadata never download the image at all.

{
  "event_id": "img-7f3a",
  "store_key": "uploads/2027/03/12/7f3a.jpg",
  "bytes": 3145728,
  "sha256": "9b1c...e4",
  "content_type": "image/jpeg"
}

The pattern's cost is 2 lifetimes to keep in step. The record expires under retention.ms; the object expires under whatever rule the store applies. If the object is deleted first, a reader replaying the topic finds a pointer to nothing, so the store's retention must be at least the topic's, and a reader must treat a missing object as a case to handle rather than a crash.

Pattern 2: chunking

When the payload must travel through the log, split it. The producer cuts the 3 MB into 4 chunks of under 1 MiB, sends each as its own record with the same key, so all 4 land in 1 partition in order, and puts a chunk id, an index and a total in the record's headers. The consumer collects chunks by id until the count matches the total, reassembles, and processes. Readers of the topic must all know the convention, and a reader that starts between chunk 2 and chunk 3 of a payload sees an incomplete set and must drop it.

def chunk_records(payload: bytes, key: str, chunk_id: str, size: int = 900_000):
    total = (len(payload) + size - 1) // size
    for index in range(total):
        headers = [
            ("chunk-id", chunk_id.encode()),
            ("chunk-index", str(index).encode()),
            ("chunk-total", str(total).encode()),
        ]
        yield key, payload[index * size:(index + 1) * size], headers

Claim check

• Log stays small; limits stay default
• Needs a second system and its retention
• Readers choose whether to fetch the bytes
• Payload can be any size

Chunking

• Everything travels through the log
• Needs a convention every reader follows
• Readers must buffer and reassemble
• Payload bounded by retention and partition throughput

Do

• Raise max.message.bytes on the 1 topic that needs it rather than message.max.bytes on the broker
• Carry a checksum with any claim check or chunk set
• Keep the store's retention at least as long as the topic's

Avoid

• Raise only the producer limit and expect the write to succeed
• Chunk across keys; chunks of 1 payload must share a partition
• Let a reader crash on a missing object or an incomplete chunk set

About this lesson

This section is part of the The Log: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.