DataFrame, Dataset, and RDD
Spark exposes 3 APIs over a single engine. They differ in what a unit of data is, how much of your code the optimizer can see, and when a mistake in your code gets caught. That last difference matters more than it sounds, because it decides whether a typo costs you a second at your desk or 40 minutes of cluster time.
| Question | RDD | DataFrame | Dataset |
|---|---|---|---|
| Unit of data | Any object | A Row with named, typed columns | An object of a class you define |
| Optimized by Catalyst | No | Yes, every step | Column steps yes, lambda steps no |
| A wrong column or field name fails | At the action, inside a task | When the line runs, before any job | Typed field access at compile time, column names when the line runs |
| Available from Python | Yes | Yes | No |
A DataFrame is a Dataset of rows
Dataset is the typed API in Spark's compiled JVM languages. You declare a class, say Order with 4 fields, and get a distributed collection of Order objects. The compiler then rejects code that reads a field the class lacks. A DataFrame is the same structure with the generic Row type standing in for your class, which is why Spark's documentation describes a DataFrame as a Dataset of Row. Python has no compile step that could check field names, so it gets DataFrames only. Every DataFrame you have written in these lessons is this untyped form: rows whose column names Spark checks when each line runs, with no compiler involved.
The typed API has a catch that mirrors the escape hatch. A Dataset step written with column expressions is optimized like any DataFrame step. A step written as a lambda over your class is opaque. Spark must rebuild each row as an object of your class before it can call the function, then convert the result back, and Catalyst cannot move a filter past that step or prune columns through it. Typing buys earlier error checks, and the lambdas still cost what lambdas cost.
A typo caught at once or 40 minutes in
Suppose a teammate types statuss for status. Here is where each of the 2 Python APIs stops them.
completed_rdd = orders.rdd.filter(lambda r: r.statuss == "Completed")
n = completed_rdd.count()
completed_df = orders.filter(F.col("statuss") == "Completed")The RDD line is accepted without complaint, because Spark does not inspect a lambda when you define it. The mistake surfaces at count(), when a task first calls the lambda on a row and Python raises an AttributeError. If that filter sits after a 40-minute stretch of work in the same job, the job runs for 40 minutes before failing. The DataFrame line fails the moment it executes, with an AnalysisException that names statuss and points to the columns that do exist, and no job ever starts.
This holds for a standard Spark session, where your program runs inside the driver. A session that talks to a remote Spark server through a thin client can defer the check until the first action or the first schema lookup.
A nightly job fails 35 minutes in with AttributeError: statuss, raised inside a task. Which API did the failing step use?
- A DataFrame filter, because column mistakes appear when tasks run
- An RDD lambda, because its field access is first tried when a task runs it
- A Dataset lambda, because typed code checks its fields at runtime
- Any of the 3, since every API checks names only at the action
Choosing between them
- Do
- Write new jobs with DataFrames so every step reaches Catalyst
- Filter and select before any .rdd call to shrink what crosses
- Use .rdd for partition counts and setup done once per partition
- Check printSchema() before typing column names into a long job
- Avoid
- Copy old RDD code line by line into a new job
- Convert to .rdd to express a filter or a sum
- Expect a plan to show work done before an RDD conversion
- Assume a typed lambda is optimized because its data is typed
One rule covers most cases. Start every job with DataFrames. Drop to an RDD for one step only when that step has no column-expression form, and do it after the filters and selects. Datasets make sense in a typed JVM codebase, where compile-time field checks are worth having, and even there the optimizer helps only their column-expression steps.
You're writing a Python job and want Spark to check column names before reading any data, and to optimize every step. Which API meets both needs?
- RDD, with each column referenced by its tuple position
- Dataset, imported into the Python session for its types
- RDD converted with toDF at the end of the chain
- DataFrame, whose column expressions Catalyst checks and optimizes
About this lesson
This section is part of the Catalyst Optimizer: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.