Compression: Bytes Versus CPU
Concepts covered: Compression & Encoding
Encodings shrink a column using what its values mean. A codec then squeezes the encoded bytes further by finding repeated byte patterns, with no idea what they represent. Every codec sits somewhere on 3 axes: how small it makes the data (ratio), how fast it restores it (decompression speed), and whether a reader can start in the middle of the output (splittability).
| Codec | Ratio | Decompression per core |
|---|---|---|
| LZ4 | about 2.1x | about 4,500MB/s |
| Snappy | about 2.1x | about 1,800MB/s |
| zstd, level 1 | about 2.9x | about 1,700MB/s |
| gzip (deflate), level 1 | about 2.7x | about 400MB/s |
Round figures on general text, in line with the zstd project's published benchmark. Speed counts the uncompressed bytes produced. Your data will give different numbers, and the ordering usually holds.
The table splits into 2 families. LZ4 and Snappy give up ratio for speed. zstd and gzip compress harder, but zstd still decompresses quickly while gzip needs around 4 times the CPU per byte decompressed. zstd also offers levels up to 22: higher levels spend more CPU on write for a better ratio, and decompression speed barely changes.
Which side of the machine is slower?
The better codec depends on where a query waits. Take a job whose scan decompresses to 600GB, running on 100 cores. Reading and decompressing overlap, so the slower of the 2 sets the pace. From object storage the cluster pulls 5GB/s in total; from local SSD it pulls 100GB/s.
| Codec | On disk | Read, object storage | Read, local SSD | Decompress, 100 cores |
|---|---|---|---|---|
| LZ4 | 286GB | 57s | 2.9s | 1.3s |
| Snappy | 286GB | 57s | 2.9s | 3.3s |
| zstd, level 1 | 207GB | 41s | 2.1s | 3.5s |
| gzip, level 1 | 222GB | 44s | 2.2s | 15s |
On object storage every codec decompresses faster than the network delivers, so the job waits on bytes. zstd finishes in roughly 41s against 57s for LZ4 because it moves 79GB less. On local SSD the bottleneck flips: reading takes 2 to 3 seconds for every codec, gzip's 15s of decompression dominates, and LZ4 finishes first at 2.9s while zstd needs 3.5s.
Splittability: can 100 cores share one file?
A cluster is fast because many cores read different parts of the data at once. A gzip-compressed CSV file breaks that, because a deflate stream can only be decoded from its beginning. Land a single 20GB gzip CSV and it expands to around 54GB, which 1 core decompresses in roughly 135 seconds while the other 99 wait. Snappy or LZ4 wrapped around a whole CSV file usually has the same problem.
Columnar files avoid this by compressing small pages inside each column chunk independently. A reader jumps to any row group through the footer and decodes it alone, so a columnar file is splittable with every codec in the table. The codec decides splittability only when it wraps an entire file.
- One core per file
- A CSV or JSON file under whole-file gzip
- A CSV file under whole-file Snappy or LZ4
- Any compressed stream without a block index
- Many cores per file
- Columnar files with any codec, split by row group
- Uncompressed CSV, split at line breaks
- Formats that compress independent, indexed blocks
A nightly job scans 3 years of cold sensor readings stored on object storage as Snappy-compressed columnar files. While it runs, the cluster's CPUs sit at 30%. Which change most likely shortens it?
- Rewrite the files with zstd so the job pulls fewer bytes from storage
- Rewrite the files with LZ4 because it decompresses faster than Snappy
- Rewrite each day as 1 gzip CSV file for a better ratio than Snappy
- Turn compression off so the CPUs stop spending time on decompression
About this lesson
This section is part of the Storage Layers: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.