Transformations vs Actions

Concepts covered: Spark Execution Model

df.filter(...).groupBy(...).agg(...) runs nothing. Spark reads no rows, starts no tasks and contacts no executor, because each of those calls is a transformation: it adds a step to a plan the driver is holding and returns at once. Twenty of them can be chained and the cluster stays idle throughout. Work starts when an action is called, such as count(), collect() or write, and the action runs the whole accumulated plan in 1 burst.

Transformations
  • filter, select, groupBy, join, withColumn
  • Return a new DataFrame at once
  • Append a step to the driver's plan
  • Free to chain, nothing runs
Actions
  • count, collect, write, show, take
  • Block until the executors finish
  • Run every transformation above them
  • Each one starts at least 1 job in the Spark UI

Why the plan waits

The gap exists so the driver sees the whole chain before committing to a plan. Seeing all of it lets the Catalyst optimizer collapse adjacent steps and push filters down into the file read. A program that reads a 2 TB transactions table, filters to 1 day and sums the amounts never loads 2 TB: the filter travels down into the scan, the Parquet reader skips every file whose date range excludes that day, and the executors read the 6 GB that match. Had filter run eagerly on the line where it was written, 2 TB would already be in memory before the planner saw the next line.

The error surfaces at the action

Because nothing runs until the action, the line that throws is almost never the line that is wrong. A UDF that fails on 1 malformed row, written 30 lines up, sits in the plan untouched by any data until the write at the bottom forces execution, and the stack trace points at the write. The same holds for slowness: an action that takes 40 minutes is paying for every transformation above it, and the expensive step, usually a join or a groupBy, is somewhere in the chain.

Actions cost different amounts

take(20) and count() are both actions, and they cost wildly different amounts. On a chain of narrow transformations, take(20) needs 20 rows, so Spark starts 1 task on 1 partition. If the first partition has not yielded 20 rows, Spark launches another job over more partitions, which is why 1 take can show as several jobs in the UI. count() needs every row, so it runs 1 task per partition. A chain that ends in take(20) and finishes in 2 seconds says nothing about the same chain ending in count(). A notebook that feels fast because every cell ends in show() can hide a job that takes an hour in production. Spark does as much work as each action demands and no more.

The cell below filters products to in-stock rows with a known category and counts them per category. Each line adds to the plan. The editor's display is the action, and when it fires, the 2 filters, the grouping and the sort run together.

Change the sort direction and run again. The plan is rebuilt from the top and executed again, because Spark kept no intermediate result. The Jobs tab shows more than 1 job for this cell, since orderBy first runs a small sampling job to pick the boundaries between sorted partitions. Every action re-runs the full chain unless the program has cached a step, which is a lesson of its own.

A notebook cell defines df = products.filter(F.col("in_stock") == 1).groupBy("category").agg(F.count(F.lit(1))) and the next cell calls df.count(). At which call does Spark first read a row from products?

An action turns the plan into tasks, and how many of those tasks run at the same moment is set by a number the program never mentions: the cores each executor was given.

About this lesson

This section is part of the Execution Model: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.