Driver, Executors, and Tasks

A SQL database runs as 1 engine: 1 process parses the statement, plans it and scans the rows, on 1 machine. Spark splits that engine into separate processes with separate responsibilities, and a Spark application is the conversation between them. Nearly every surprising thing Spark does, from a collect() that crashes to a cluster of 100 machines running slower than a laptop, is a statement about which process was asked to do what.

The work travels to the data

The driver never scans the data. It reads the code, builds a plan, ships small task descriptions to the executors and waits for them to report back. The data sits in distributed storage such as S3 or HDFS. Each task reads its own slice straight into the executor that runs it, and the rows stay there unless the program asks for them elsewhere. A task description is a few kilobytes, so sending 8,000 of them costs almost nothing. Pulling a 2 TB table to 1 machine would take close to half an hour over a 10 Gbit link, and the table would fit in no driver's memory in the first place.

collect() is the most common way rows reach the driver, alongside toPandas() and the building of a broadcast table. What makes collect() dangerous is that it pulls every row: each executor sends its whole share into the driver's memory. Spark aborts a job whose collected result passes spark.driver.maxResultSize, 1 GB by default, and once that limit is raised, a driver that receives more than it can hold dies of an out-of-memory error that reads as if the cluster failed. The cluster was fine. The wrong process was handed the rows.

What a task is

A task is the unit of work the driver ships: 1 chain of operations applied to 1 slice of the data, run by 1 thread on 1 executor. The driver creates the tasks, records which executor took each one, notices when 1 fails, and collects each task's report when it finishes. An executor runs several tasks at once, each in its own thread, and knows nothing about the tasks running on other executors. The split of data into slices is the next section's subject, and it decides how many tasks there are.

SQL engine

• 1 process plans the query and scans the rows
• All rows sit on 1 machine's disks
• Faster by buying a bigger machine
• The planner reads the rows itself

Spark

• The driver plans, the executors scan
• Rows spread across many machines
• Faster by adding executors
• The planner never touches the data

Which process is busy

Because separate processes share the job, the first question about a slow Spark job is which one is busy. A driver at 100% CPU with idle executors means the program asked the driver to do data work, usually a collect() or a large broadcast being serialised. Executors at 100% with a long queue of waiting tasks mean the job needs more capacity, or was given too little memory and is spilling to disk. Executors that never arrived mean the cluster manager had nothing to grant, and the job runs on the 2 executors it received instead of the 20 it asked for. Naming the process is most of the diagnosis, and the Spark UI's Executors tab shows all 3 states in 1 glance.

A nightly job reads 400 GB of order lines, aggregates them and writes a summary. It normally finishes in 20 minutes with 20 executors; tonight it has run for 3 hours. The Executors tab shows 2 executors alive, each at 100% CPU, and the driver idle. Which process is the bottleneck?

• The cluster manager, which granted 2 executors instead of 20
• The driver, because it is coordinating too many tasks
• The executors, because they are at 100% CPU
• The data, because 400 GB is too much for Spark

Executors do the work, and the unit they work in is smaller than a table. Spark cuts every dataset into partitions and assigns 1 task to each, and that split decides how many executors can be busy at the same moment.

About this lesson

This section is part of the Execution Model: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.