The Cost of CSV and JSON

Concepts covered: Serialization

Text formats store every value as characters and every row as a line. That makes them readable by anything, and expensive for Spark in 4 ways: more bytes, parsing work, missing types and, once gzipped, a single task per file.

More bytes for the same event

Take a single event where user_id is 48211 and country is BR, with event_type set to click and dur_ms at 5120. Written as a JSON line, the record repeats every key name:

{"user_id":48211,"country":"BR","event_type":"click","dur_ms":5120}

As a CSV line it is 48211,BR,click,5120: the key names move into a header row, but the number 48211 still takes 5 characters at 1 byte each. In Parquet the same event takes about 10 bytes before compression: 4 for each integer and 1 for each dictionary code.

Parsing every character

To find dur_ms in a CSV line, the parser walks every character up to the 3rd comma, because nothing in the file records where a field starts. Then it converts the digits 5120 into an integer a character at a time, checking that each one is a digit. A JSON parser does the same work on braces and quotes, and matches every key name against the schema before it knows which field a value belongs to. Skipping a field on text still means reading past it, so this work grows with the full width of every line.

No types, so Spark guesses or reads strings

A CSV file carries no types. Read it without a schema and every column arrives as a string. Turn on inferSchema and Spark makes an extra pass over the data to guess types before the real read starts. JSON sources infer a schema by default, with the same extra pass. Passing an explicit schema avoids both costs, and Parquet never needs one because its footer already holds the types.

Compressed text is one task per file

Text files are often gzipped to save space. A gzip stream can only be decompressed from its first byte, so Spark cannot split a .csv.gz file among tasks: a 20 GB events.csv.gz becomes 1 partition read by 1 core while the rest of the cluster waits. A Parquet file compresses page by page and still splits by row group, which the next section explains.

The bill, in bytes and hours

Here is the running example priced out. The events history is 6 TB of CSV with 30 columns. Its Parquet copy is 1.5 TB. The daily report reads 3 columns, so on Parquet it fetches about 150 GB. The cluster has 100 cores, and suppose each core gets through 25 MB/s of input, 2.5 GB/s across the cluster. The report runs 10 times a day.

Per dayCSVParquet
Bytes read per run6 TB150 GB
Time per run at 2.5 GB/s40 min1 min
Runs per day1010
Cluster time per day6 h 40 min10 min

Converting costs 1 full read of the CSV, 40 minutes, plus the time to write 1.5 TB. Against 6 h 30 min of cluster time saved every day, that cost is repaid on day one. The 25 MB/s rate is also generous to CSV: decoding binary Parquet columns is faster per byte than parsing text, so the real gap is wider than the table shows.

Each day's CSV is parsed once by the conversion job, and every report after that reads Parquet columns.
  • Landing CSV (source, CSV files, daily)
  • Convert (transform, Spark job, explicit schema)
  • events (storage, Parquet + Snappy)
  • Daily report (consumer, Spark, 3 columns)
  • Landing CSV -> Convert
  • Convert -> events
  • events -> Daily report
schema = "event_id BIGINT, user_id INT, country STRING, event_type STRING, dur_ms INT"

raw = (
    spark.read
    .option("header", True)
    .schema(schema)
    .csv("/landing/events/2024-03-15/")
)
raw.write.mode("overwrite").parquet("/data/events/2024-03-15/")

A nightly job reads a single 30 GB clicks.csv.gz file on a 64-core cluster and runs for 2 hours. What explains most of the runtime?

About this lesson

This section is part of the Storage Formats: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.