Compression with Snappy

Concepts covered: Configuration Tuning

Spark compresses every Parquet file it writes with Snappy unless told otherwise. The setting is spark.sql.parquet.compression.codec, its default is snappy, and the codec shows up in the file names as .snappy.parquet.

Where Snappy sits in the write path

A reader runs the same steps backwards: decompress the page, then decode it.
  • Column values (source, one page's worth)
  • Encode (transform, dictionary, run-length)
  • Compress (transform, Snappy)
  • Page on disk (storage, about 1 MB before compression)
  • Column values -> Encode
  • Encode -> Compress
  • Compress -> Page on disk

Compression runs on each page after encoding, so Snappy squeezes data that dictionary codes and run lengths have already shrunk. Because every page is compressed on its own, a task assigned the second row group of a file starts decompressing at that row group's first page, with no need for the bytes before it, and so a Snappy Parquet file splits across tasks where a gzipped CSV cannot.

Fewer bytes to fetch

Snappy finds repeated byte sequences and replaces later copies with short references to earlier ones. In the running example, the encoded but uncompressed Parquet copy of events would be 2.4 TB, and Snappy brings it to the 1.5 TB in the bill from the CSV section. The report's 3 columns shrink from 240 GB to 150 GB, and at the cluster's 2.5 GB/s, fetching them drops from 96 seconds to 60.

CPU on every page

Every compressed page must be decompressed before Spark can use a value in it, and every page written must first be compressed. Google, which wrote Snappy, publishes its speed on a single core of the processor it measured:

At that decompression rate, 100 cores turn 150 GB of compressed pages back into 240 GB of encoded values in about 5 seconds. The report pays about 5 seconds of CPU to save 36 seconds of reading. On the write side, compressing the 2.4 TB copy takes around 2.7 core-hours of CPU, paid once by the conversion job.

Per report runUncompressed ParquetSnappy Parquet
Bytes fetched240 GB150 GB
Fetch time at 2.5 GB/s96 s60 s
Decompression time0 sabout 5 s
Total96 sabout 65 s

When the trade turns

Snappy assumes the CPU is waiting on storage. That assumption fails when a binary column holds images or encrypted blobs, which are already compressed: there are no repeats left to find, so Snappy spends CPU and saves nothing. It also fails for a table that is rarely read but kept for years, where size matters more than speed and slower codecs that pack tighter can win. For tables Spark scans every day, the default holds.

What Snappy buys
  • Fewer bytes fetched for every column read
  • 2.4 TB stored as 1.5 TB in the running example
  • Pages that decompress independently, so files stay splittable
  • Decompression fast enough that it rarely becomes the slow step
What Snappy costs
  • CPU on every page read and every page written
  • A weaker ratio than slower codecs reach
  • Wasted work on data that is already compressed
  • Compression at half its decompression speed, paid at write time
spark.conf.get("spark.sql.parquet.compression.codec")

events.write.option("compression", "snappy").parquet("/data/events_copy/")

A report reads 3 Snappy-compressed Parquet columns and spends most of its time waiting on storage. A teammate proposes rewriting the table uncompressed to save CPU. What happens to the report?

About this lesson

This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.