Choosing a Codec
Concepts covered: Cost Optimization
Spark compresses Parquet with Snappy unless told otherwise. The codec runs page by page, after dictionary and run-length encoding have already shrunk the data, so any codec leaves the file splittable. Every codec charges CPU twice, once to compress at write and again to decompress at every read, and pays it back in bytes stored and bytes moved. Choosing one means deciding which of those your table can afford.
(trips
.write
.option('compression', 'zstd')
.mode('overwrite')
.parquet('/lake/trips_zstd'))
spark.conf.set('spark.sql.parquet.compression.codec', 'zstd')The writer option covers a single write and overrides the session setting, which covers every Parquet write after it. Each column chunk records its own codec, so a directory can mix Snappy and ZSTD files and still read as a single table, and switching codecs never forces a rewrite of old data.
Four codecs on the same benchmark
| Codec | Ratio | Compress | Decompress |
|---|---|---|---|
| snappy | 2.1x | 550MB/s | 1,750MB/s |
| lz4 | 2.1x | 740MB/s | 4,500MB/s |
| zstd, level 1 | 2.9x | 530MB/s | 1,700MB/s |
| gzip (zlib), level 1 | 2.7x | 95MB/s | 400MB/s |
Rounded from the single-core Silesia corpus table in the zstd project's README. Figures move with each release and CPU, and Parquet pages compress differently from raw text, so read them as relative speeds.
Snappy and LZ4 trade ratio for speed. ZSTD at a low level comes close to gzip's ratio while decompressing about 4 times faster, which leaves gzip few cases where it wins. Parquet's own defaults sit further along the curve: gzip runs at level 6, and ZSTD at level 3 unless parquet.compression.codec.zstd.level says otherwise.
Measuring on your own table
The benchmark compresses raw text. Your columns arrive already dictionary-encoded and will compress differently, so measure on a sample:
sample = spark.read.parquet('/lake/trips').sample(fraction=0.01, seed=7)
for codec in ['snappy', 'zstd', 'gzip']:
(sample
.write
.option('compression', codec)
.mode('overwrite')
.parquet(f'/tmp/codec_test/{codec}'))Each write is its own action, so the sample is recomputed 3 times; the fixed seed returns the same 1% every time. Compare the 3 directory sizes and time a scan of each. Say the results come back like this, for a table holding 2,000GB of encoded pages before compression:
| Codec | 1% sample | Full table | Decompress core-seconds per full scan |
|---|---|---|---|
| snappy | 9.0GB | 900GB | ~1,170 |
| zstd | 6.6GB | 660GB | ~1,200 |
| gzip | 6.3GB | 630GB | ~5,120 |
The last column divides 2,000GB by each codec's benchmark decompression speed. ZSTD stores 240GB less than Snappy and moves 240GB less over the network on every full scan, for roughly 3% more decompression CPU. Gzip saves another 30GB of storage and costs over 4 times ZSTD's CPU on every read. Scanned 50 times a day, that gap comes to roughly 196,000 core-seconds, roughly 54 core-hours, daily.
Picking by read pattern
- Hot table, many reads a day: Decompression cost per byte is close, and ZSTD moves fewer bytes, which wins when scans wait on storage.
- CPU-bound job on fast local disks: Bytes are cheap to fetch, so the codec that spends the least CPU wins.
- Archive written once, read rarely: The slow compression is paid once at write, and reads still decompress at ZSTD speed.
- Files read by other engines: Confirm each consumer can decode the codec before switching.
A dashboard table is scanned in full around 200 times a day from object storage, and its scans wait mostly on the network. It uses Snappy. The storage team proposes gzip for smaller files. Going by the benchmark above, which choice fits best?
- gzip, since smaller files cut network time on all 200 daily scans
- lz4, since it has the fastest decompression in the table
- zstd at a low level, since it nears gzip's ratio at Snappy-like decompression speed
- Keep Snappy, since changing codecs means rewriting every existing file
About this lesson
This section is part of the Storage Formats: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.