Errors That Surface at the Action
Concepts covered: Spark Execution Model
A transformation line returns a new DataFrame and runs nothing. trips.filter(F.col("status") == "completed") records one step, hands back an object that describes the result, and the driver sends no task to any executor. The recorded chain turns into a job only when an action asks for a result. So when an error's cause sits inside the data, the error stays hidden until a task reads that data, and tasks read data only during the action.
The bug on line 2, the crash on line 4
trips = spark.read.parquet("s3://rides/trips/2024-03-14/")
priced = trips.withColumn("fare_cents", F.col("fare_text").cast("int"))
per_city = priced.groupBy("city").agg(F.sum("fare_cents").alias("fare_total"))
per_city.collect()The column fare_text holds strings such as "1250", and one row somewhere in the day's 400 files holds "12.50USD". Line 2 carries the bug, because a cast from that string to an integer cannot succeed. Spark cannot see the bad value at line 2: the footer of each file tells Spark that fare_text exists and is a string, and nothing more. So line 2 records the cast and returns. Line 4 builds the job, and because collect() returns fare_total, every row's fare_cents has to be computed. The executors start reading the 400 files, and 18 minutes in, a task reaches the bad row. With ANSI mode on, the default since Spark 4.0, a cast that can't parse its input raises CAST_INVALID_INPUT instead of returning null. The failed task takes the job down with it, and the traceback ends at per_city.collect(), the line the driver was executing when the failure came back.
| Mistake | What Spark can know when the line runs | Where it surfaces |
|---|---|---|
| Column name misspelled | The file footer lists every column | The transformation line, as an AnalysisException |
| Path that does not exist | read() lists the files at that path | The read line |
| Text that cannot be cast | Only the column's type, never its values | The action, when a task reads the row |
| A zero in a column you divide by | Only the column's type | The action, when a task reaches that row |
| A file deleted after read() ran | The listing from when read() ran | The action, when a task opens the file |
The table above follows one pattern. Spark checks what it can learn from metadata on the line itself, and leaves everything that needs a row until a task has that row. A misspelled column fails on its own line with cannot resolve, because the schema arrived with the file listing. A value that breaks a cast, a zero that breaks a division, or a file that vanished between the listing and the job can be found only by a task, and a task runs only inside an action.
The cell below records 2 transformations and ends with count(). Run it and note that the only output is the one number the action returns. The filter ran inside the job, and because count() needs no profit_cents value, Spark never computed one. The DataFrame priced itself never held a row.
Every action follows this rule. count() and collect() each start a job that reads every partition, and so does a write, so whichever of them first runs a broken chain is the line the traceback ends at. A step placed after the broken one gives no protection either: a filter on a cast column computes the cast for each row before it can decide whether to keep that row.
A nightly job runs the program below, and one row in the source holds qty_text equal to "n/a". ANSI mode is on. Which line does the traceback end at?
stock = spark.read.parquet("s3://wh/stock/")
typed = stock.withColumn("qty", F.col("qty_text").cast("int"))
low = typed.filter(F.col("qty") < 10)
low.count()- Line 2, because the cast is the step that cannot handle the value n/a.
- Line 4, because count() is the action that starts the job whose task reads the bad row.
- Line 1, because reading the files is when Spark first touches the data.
- No line, because a cast that fails returns null and the filter then drops the row.
About this lesson
This section is part of the Lazy Evaluation: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.