DataFrames over RDDs

Every transformation you write adds a step to a plan, and nothing runs until an action. What Spark can do with that plan before the action depends on what each step is made of. A step is either an RDD operation, which takes a Python function, or a DataFrame operation, which takes column expressions. Written both ways, one business question can look almost identical on the page.

Here is a single question, total profit of completed orders per region, in both styles. orders_rdd holds tuples of (order_id, status, region, profit), and orders is a DataFrame with those same 4 columns.

rdd_result = (
    orders_rdd
    .filter(lambda row: row[1] == "Completed")
    .map(lambda row: (row[2], row[3]))
    .reduceByKey(lambda a, b: a + b)
)

df_result = (
    orders
    .filter(F.col("status") == "Completed")
    .groupBy("region")
    .agg(F.sum("profit").alias("total_profit"))
)

Lambdas hide their logic, expressions show it

Run either chain and you get the same totals. The difference is what Spark knows before running them. To Spark, lambda row: row[1] == "Completed" is a compiled Python function. Spark can call it on an element and get a value back, and that is all it can do. It can't tell which field the function reads, that the test is an equality, or that position 1 holds a status string. The DataFrame filter hands Spark the expression itself: = applied to the column status and the literal Completed.

If Spark can read a step, it can change that step. The DataFrame chain is built from expressions over named columns, so Spark passes the whole plan through its optimizer, called Catalyst, before scheduling the first task. Catalyst notices that order_id is never used and leaves it unread, and it chooses how the grouping is carried out. The RDD chain gets none of this. Spark runs each lambda exactly where you placed it, on every field of every element, in the written order.

RDD: Spark follows your steps

• Each step is a function Spark can call but cannot inspect
• Steps run in the order written, with nothing reordered
• Every field of every element travels through every step
• Elements are Python objects handled by Python code

DataFrame: Spark plans your steps

• Each step is an expression over named, typed columns
• Catalyst may reorder or merge steps when the rows stay the same
• Columns no step reads are cut from the plan
• Rows stay in Spark's binary format inside its engine

Spark receives this step in an RDD chain. Before any task runs, what can Spark determine about it?

big_orders = orders_rdd.filter(lambda row: row[3] > 1000)

• That it reads field 3 and compares it to 1000, so the test can move earlier
• That field 3 holds a number, since a comparison with 1000 requires one
• Only that it is a function to call on each element; what it reads stays hidden
• That the other 3 fields go unused, so it can drop them before this step

Laziness gives the optimizer its chance

You already know Spark waits for an action before doing any work. That wait is what gives Catalyst room to act. When you call collect() or count(), Spark holds the full chain from the read to the result, so it can optimize all of it at once. With DataFrames, a messy chain of 10 steps and a tidy chain of 4 can come out as the same plan. With RDDs, 10 steps run as 10 steps.

Run the DataFrame version on the practice orders table. The extra filter drops orders that have no region, which would otherwise form a group of their own. Then change "Completed" to "completed". The query still runs, but every group disappears, because string comparison is case-sensitive.

About this lesson

This section is part of the Catalyst Optimizer: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.