The Cost of inferSchema

Concepts covered: Schema Definition

A CSV file carries no types. Every value is text, and the only metadata is an optional header line of column names. When you read one, every column still needs a type. Spark can leave every column as string or use a schema you declare, and neither costs anything extra, or it can infer the types, which costs a full extra pass over the data.

raw = spark.read.option("header", "true").csv("/lake/orders/")

inferred = (
    spark.read.option("header", "true")
    .option("inferSchema", "true")
    .csv("/lake/orders/")
)

With inferSchema set, Spark parses every row of every file and, for each column, tracks the narrowest type that still fits every value seen so far: integer until a decimal point appears, double until a letter appears, string after that. That's a complete scan of the input, and it runs when .csv(path) is called, before you have asked for anything. You'll see a job appear in the UI with no action in your code. When you do call an action, the real job reads the same bytes a second time. On a 500GB directory you have read 1TB to get an answer that needed 500GB.

• 2x
• 1 line

Declaring the schema

orders_schema = "order_id INT, status STRING, region STRING, profit DOUBLE"

orders = (
    spark.read.option("header", "true")
    .schema(orders_schema)
    .csv("/lake/orders/")
)

The schema string names each column and its type in the same form you would use in a CREATE TABLE. With it in place, the read opens nothing until an action runs, and then the parser converts each value straight into the declared type. The same schema can be built as a StructType of StructField objects, which you reach for when a field is nested. Either way the cost of learning the types drops to 0 bytes, and the types are the ones you chose rather than the ones a scan happened to find.

Inference also guesses

The extra scan would be tolerable if it always produced the right schema. It produces the schema that fits the data on that day. A postal code column holding 02134 infers as an integer and comes back as 2134, with the leading zero gone for good. An amount column that is numeric in a billion rows and holds the text N/A in 1 row infers as string, and the arithmetic you write downstream runs on a string column: depending on your settings it either fails at the action, after the job has run for an hour, or quietly turns the bad values into null. A column that is empty in every row infers as string, and the day it fills with numbers the schema changes under every job that reads it. A declared schema keeps the types you chose: a value that doesn't parse becomes null by default, and .option("mode", "FAILFAST") makes the read fail at the first malformed row.

JSON reads pay the same pass. spark.read.json(path) infers by default, so a JSON read without .schema(...) always pays the full pass. Both readers accept a samplingRatio option that infers from a fraction of the rows, which shortens the scan and makes the guess less reliable at the same time; a declared schema avoids both problems.

Do

• Declare the schema for every CSV and JSON read in production
• Keep the schema next to the job so a change to it is reviewed
• Infer once on a sample, then paste the result as the declaration
• Set mode to FAILFAST so a malformed row stops the read and sends you to the source

Avoid

• Pay a 500GB scan to learn 6 column types
• Let today's data decide tomorrow's types
• Store postal codes or IDs with leading zeros as inferred integers
• Expect a sampled inference to see the 1 row that breaks the type

A job reads a 2TB CSV directory with inferSchema set to true, filters to 1 region, and calls count(). Roughly how many bytes does the job read in total, and when does the first scan happen?

• 2TB in total, because the inference pass is reused as the data for the count
• 4TB in total: 2TB when .csv(path) is called to infer the types, and 2TB again when count() runs
• 2TB plus the header line, with inference running lazily when count() is called
• 4TB in total, both scans triggered by count() because nothing runs before an action

About this lesson

This section is part of the Partitioning: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.