Compression on the Wire

Concepts covered: Batching and compression

The batch is the unit of compression

compression.type on the producer takes none for no compression, or names one of the codecs gzip, snappy, lz4 and zstd. The producer compresses records into the batch as it appends them, so the unit of compression is the batch. A codec earns its ratio by finding repeats, and in event data the repeats sit across records rather than inside one: 100 order events share the same field names, the same handful of store ids and the same timestamp prefix. A batch of 100 compresses well. A batch of 1 has nothing to repeat against, and under gzip it comes out larger than it went in, because the codec writes a 10-byte header and an 8-byte trailer of its own around bytes it couldn't shrink.

The model for the table: each event is 220 bytes, 150 of them structure repeated from one event to the next and 70 that differ. The codec writes the first copy of the structure in full and each later repeat as a 3-byte reference, plus 18 bytes of gzip framing per batch.

Real data beats the model, because the 70 differing bytes repeat partly too: 2 events a second apart share most of their timestamp digits, and a store id that appears in 30 events of 100 is a repeat as well. Measured ratios on event streams run higher than 3 and keep climbing with batch size until the codec's window of memory fills.

Where the cost lands

Compression spends CPU on the producer as the batch is built, and the compressed batch is what travels: the produce request is smaller, the leader appends the compressed bytes as they arrived, followers replicate those same bytes, the segment on disk holds them, and a fetch returns them. Each reader of the topic spends CPU to decompress, once per batch per reader. The byte saving therefore multiplies: a batch that shrinks 3 times saves 3 times on the producer's network, 3 times on every replication copy, 3 times on disk for the whole retention window, and 3 times on every consumer fetch. The CPU cost stays with the producer and each reader.

The topic setting compression.type defaults to producer, which tells the broker to keep whatever codec the producer used. Set it to a named codec instead and the broker decompresses every batch that arrives in a different codec and recompresses it, which moves the CPU cost onto the broker, the one machine in the path that is hardest to scale out. The 1 reason to do it is a fleet of producers you do not control sending uncompressed data to a topic whose disk you pay for.

compression.type=zstd
compression.zstd.level=3
linger.ms=20
batch.size=65536

Why batch size changes the answer

Because the ratio comes from repeats across records, the same codec on the same data gives a different answer at every batch size. The batch size depends on linger.ms and batch.size as well as on the record rate per partition from the previous section. A producer at linger.ms=0 shipping 4-record batches may measure gzip at 1.4 times and conclude compression isn't worth the CPU; the same producer at linger.ms=20 ships 40-record batches and measures 3 times. There's a second coupling. The producer checks batch.size against the batch's estimated compressed size, using a ratio it learns from recent batches, so a 16384-byte ceiling with a 3 times ratio holds around 48 KB of raw records. Turning compression on makes batches hold more records, which raises the ratio further. The growth stops when linger expires first or the record rate can't fill the batch.

About this lesson

This section is part of the Producers: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.