The Cost of CSV and JSON
Concepts covered: Serialization
Text formats store every value as characters and every row as a line. That makes them readable by anything, and expensive for Spark in 4 ways: more bytes, parsing work, missing types and, once gzipped, a single task per file.
More bytes for the same event
Take a single event where user_id is 48211 and country is BR, with event_type set to click and dur_ms at 5120. Written as a JSON line, the record repeats every key name:
{"user_id":48211,"country":"BR","event_type":"click","dur_ms":5120}As a CSV line it is 48211,BR,click,5120: the key names move into a header row, but the number 48211 still takes 5 characters at 1 byte each. In Parquet the same event takes about 10 bytes before compression: 4 for each integer and 1 for each dictionary code.
- 67 bytes
- 38 bytes
- 19 bytes
Parsing every character
To find dur_ms in a CSV line, the parser walks every character up to the 3rd comma, because nothing in the file records where a field starts. Then it converts the digits 5120 into an integer a character at a time, checking that each one is a digit. A JSON parser does the same work on braces and quotes, and matches every key name against the schema before it knows which field a value belongs to. Skipping a field on text still means reading past it, so this work grows with the full width of every line.
No types, so Spark guesses or reads strings
A CSV file carries no types. Read it without a schema and every column arrives as a string. Turn on inferSchema and Spark makes an extra pass over the data to guess types before the real read starts. JSON sources infer a schema by default, with the same extra pass. Passing an explicit schema avoids both costs, and Parquet never needs one because its footer already holds the types.
Compressed text is one task per file
Text files are often gzipped to save space. A gzip stream can only be decompressed from its first byte, so Spark cannot split a .csv.gz file among tasks: a 20 GB events.csv.gz becomes 1 partition read by 1 core while the rest of the cluster waits. A Parquet file compresses page by page and still splits by row group, which the next section explains.
The bill, in bytes and hours
Here is the running example priced out. The events history is 6 TB of CSV with 30 columns. Its Parquet copy is 1.5 TB. The daily report reads 3 columns, so on Parquet it fetches about 150 GB. The cluster has 100 cores, and suppose each core gets through 25 MB/s of input, 2.5 GB/s across the cluster. The report runs 10 times a day.
| Per day | CSV | Parquet |
|---|---|---|
| Bytes read per run | 6 TB | 150 GB |
| Time per run at 2.5 GB/s | 40 min | 1 min |
| Runs per day | 10 | 10 |
| Cluster time per day | 6 h 40 min | 10 min |
Converting costs 1 full read of the CSV, 40 minutes, plus the time to write 1.5 TB. Against 6 h 30 min of cluster time saved every day, that cost is repaid on day one. The 25 MB/s rate is also generous to CSV: decoding binary Parquet columns is faster per byte than parsing text, so the real gap is wider than the table shows.
- Landing CSV (source, CSV files, daily)
- Convert (transform, Spark job, explicit schema)
- events (storage, Parquet + Snappy)
- Daily report (consumer, Spark, 3 columns)
- Landing CSV -> Convert
- Convert -> events
- events -> Daily report
schema = "event_id BIGINT, user_id INT, country STRING, event_type STRING, dur_ms INT"
raw = (
spark.read
.option("header", True)
.schema(schema)
.csv("/landing/events/2024-03-15/")
)
raw.write.mode("overwrite").parquet("/data/events/2024-03-15/")A nightly job reads a single 30 GB clicks.csv.gz file on a 64-core cluster and runs for 2 hours. What explains most of the runtime?
- The CSV holds too many columns, so executors run out of memory while parsing it
- Spark infers the schema of a gzip file twice, once for each compression block
- Gzip cannot be split, so 1 task reads and parses all 30 GB while 63 cores sit idle
- Text values arrive as strings, so every aggregate adds a shuffle to convert them
About this lesson
This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.