Choosing a Codec

Concepts covered: Cost Optimization

Spark compresses Parquet with Snappy unless told otherwise. The codec runs page by page, after dictionary and run-length encoding have already shrunk the data, so any codec leaves the file splittable. Every codec charges CPU twice, once to compress at write and again to decompress at every read, and pays it back in bytes stored and bytes moved. Choosing one means deciding which of those your table can afford.

(trips
    .write
    .option('compression', 'zstd')
    .mode('overwrite')
    .parquet('/lake/trips_zstd'))

spark.conf.set('spark.sql.parquet.compression.codec', 'zstd')

The writer option covers a single write and overrides the session setting, which covers every Parquet write after it. Each column chunk records its own codec, so a directory can mix Snappy and ZSTD files and still read as a single table, and switching codecs never forces a rewrite of old data.

Four codecs on the same benchmark

CodecRatioCompressDecompress
snappy2.1x550MB/s1,750MB/s
lz42.1x740MB/s4,500MB/s
zstd, level 12.9x530MB/s1,700MB/s
gzip (zlib), level 12.7x95MB/s400MB/s

Rounded from the single-core Silesia corpus table in the zstd project's README. Figures move with each release and CPU, and Parquet pages compress differently from raw text, so read them as relative speeds.

Snappy and LZ4 trade ratio for speed. ZSTD at a low level comes close to gzip's ratio while decompressing about 4 times faster, which leaves gzip few cases where it wins. Parquet's own defaults sit further along the curve: gzip runs at level 6, and ZSTD at level 3 unless parquet.compression.codec.zstd.level says otherwise.

Measuring on your own table

The benchmark compresses raw text. Your columns arrive already dictionary-encoded and will compress differently, so measure on a sample:

sample = spark.read.parquet('/lake/trips').sample(fraction=0.01, seed=7)

for codec in ['snappy', 'zstd', 'gzip']:
    (sample
        .write
        .option('compression', codec)
        .mode('overwrite')
        .parquet(f'/tmp/codec_test/{codec}'))

Each write is its own action, so the sample is recomputed 3 times; the fixed seed returns the same 1% every time. Compare the 3 directory sizes and time a scan of each. Say the results come back like this, for a table holding 2,000GB of encoded pages before compression:

Codec1% sampleFull tableDecompress core-seconds per full scan
snappy9.0GB900GB~1,170
zstd6.6GB660GB~1,200
gzip6.3GB630GB~5,120

The last column divides 2,000GB by each codec's benchmark decompression speed. ZSTD stores 240GB less than Snappy and moves 240GB less over the network on every full scan, for roughly 3% more decompression CPU. Gzip saves another 30GB of storage and costs over 4 times ZSTD's CPU on every read. Scanned 50 times a day, that gap comes to roughly 196,000 core-seconds, roughly 54 core-hours, daily.

Picking by read pattern

A dashboard table is scanned in full around 200 times a day from object storage, and its scans wait mostly on the network. It uses Snappy. The storage team proposes gzip for smaller files. Going by the benchmark above, which choice fits best?

About this lesson

This section is part of the Storage Formats: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.