Compression with Snappy
Concepts covered: Configuration Tuning
Spark compresses every Parquet file it writes with Snappy unless told otherwise. The setting is spark.sql.parquet.compression.codec, its default is snappy, and the codec shows up in the file names as .snappy.parquet.
Where Snappy sits in the write path
- Column values (source, one page's worth)
- Encode (transform, dictionary, run-length)
- Compress (transform, Snappy)
- Page on disk (storage, about 1 MB before compression)
- Column values -> Encode
- Encode -> Compress
- Compress -> Page on disk
Compression runs on each page after encoding, so Snappy squeezes data that dictionary codes and run lengths have already shrunk. Because every page is compressed on its own, a task assigned the second row group of a file starts decompressing at that row group's first page, with no need for the bytes before it, and so a Snappy Parquet file splits across tasks where a gzipped CSV cannot.
Fewer bytes to fetch
Snappy finds repeated byte sequences and replaces later copies with short references to earlier ones. In the running example, the encoded but uncompressed Parquet copy of events would be 2.4 TB, and Snappy brings it to the 1.5 TB in the bill from the CSV section. The report's 3 columns shrink from 240 GB to 150 GB, and at the cluster's 2.5 GB/s, fetching them drops from 96 seconds to 60.
CPU on every page
Every compressed page must be decompressed before Spark can use a value in it, and every page written must first be compressed. Google, which wrote Snappy, publishes its speed on a single core of the processor it measured:
- 500 MB/s
- 250 MB/s
At that decompression rate, 100 cores turn 150 GB of compressed pages back into 240 GB of encoded values in about 5 seconds. The report pays about 5 seconds of CPU to save 36 seconds of reading. On the write side, compressing the 2.4 TB copy takes around 2.7 core-hours of CPU, paid once by the conversion job.
| Per report run | Uncompressed Parquet | Snappy Parquet |
|---|---|---|
| Bytes fetched | 240 GB | 150 GB |
| Fetch time at 2.5 GB/s | 96 s | 60 s |
| Decompression time | 0 s | about 5 s |
| Total | 96 s | about 65 s |
When the trade turns
Snappy assumes the CPU is waiting on storage. That assumption fails when a binary column holds images or encrypted blobs, which are already compressed: there are no repeats left to find, so Snappy spends CPU and saves nothing. It also fails for a table that is rarely read but kept for years, where size matters more than speed and slower codecs that pack tighter can win. For tables Spark scans every day, the default holds.
- What Snappy buys
- Fewer bytes fetched for every column read
- 2.4 TB stored as 1.5 TB in the running example
- Pages that decompress independently, so files stay splittable
- Decompression fast enough that it rarely becomes the slow step
- What Snappy costs
- CPU on every page read and every page written
- A weaker ratio than slower codecs reach
- Wasted work on data that is already compressed
- Compression at half its decompression speed, paid at write time
spark.conf.get("spark.sql.parquet.compression.codec")
events.write.option("compression", "snappy").parquet("/data/events_copy/")A report reads 3 Snappy-compressed Parquet columns and spends most of its time waiting on storage. A teammate proposes rewriting the table uncompressed to save CPU. What happens to the report?
- It speeds up, because decompression was the slowest step of the scan
- It slows down, because the scan fetches more bytes to save a few seconds of CPU
- It stays the same, because encoding already removed every repeat Snappy could find
- It fails, because Spark cannot split uncompressed Parquet files across tasks
About this lesson
This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.