Where the Driver Lives

Client mode and cluster mode

The driver is the process that splits your job into stages, hands tasks to executors and gathers results, so where it runs decides what survives a disconnect and where results land. Spark offers 2 places for it, chosen by the deploy mode.

In client mode, the default, the driver runs inside the spark-submit process on the machine you launched from: your laptop, a shared gateway box, a notebook server. In cluster mode, spark-submit asks the cluster manager to start the driver on one of the cluster's own nodes, and the driver runs there from then on while spark-submit only reports on it.

Client mode: the driver sits outside the cluster, so task launches and status updates cross the link between your machine and the executors, and so does every collect() result.

Launch machine (source, spark-submit and driver)
Cluster manager (custom, grants executors)
Executor 1 (transform, 4 slots)
Executor 2 (transform, 4 slots)

Launch machine -> Cluster manager
Launch machine -> Executor 1
Launch machine -> Executor 2

Outside the cluster: Launch machine
Inside the cluster: Cluster manager, Executor 1, Executor 2

Cluster mode: spark-submit hands the application to the cluster manager, which starts the driver on a cluster node next to its executors.

Launch machine (source, spark-submit only)
Cluster manager (custom, starts the driver)
Driver node (transform, driver)
Executor 1 (transform, 4 slots)
Executor 2 (transform, 4 slots)

Launch machine -> Cluster manager
Cluster manager -> Driver node
Driver node -> Executor 1
Driver node -> Executor 2

Outside the cluster: Launch machine
Inside the cluster: Cluster manager, Driver node, Executor 1, Executor 2

Question	Client mode	Cluster mode
Where the driver runs	The machine that ran spark-submit	A node the cluster manager picks
Where `collect()` rows land	Your machine's memory	That node's memory
Where `print()` output shows	Your terminal or notebook	The driver's log on that node
If the launch machine shuts down	The driver dies and the job is lost	The job keeps running
Typical use	Shells and notebooks, which need a driver where you type	Scheduled and long-running production jobs

You run a 40-minute job from a notebook on your laptop, so the driver runs in client mode. At minute 25 the laptop's battery dies. What happens to the job?

It carries on, and the results wait on the cluster until the laptop is back
The application ends, its executors are released, and the job must start over
The cluster manager moves the driver to a cluster node and the job finishes
Only the running stage is lost, and finished stages are kept for the next launch

The link the driver talks over

Client mode's cost is distance. The driver sends a message to launch every task and receives 1 back when it finishes, so a 400-task stage puts at least 800 messages on the link between your machine and the cluster. A collect() that returns 2GB crosses the same link: over a 50 megabit-per-second VPN that is more than 5 minutes of transfer, where a driver inside the cluster's network would take seconds.

A third arrangement runs on a single machine: --master local[4] puts the driver and 4 task threads in 1 process, so your slots are 4 threads, and shuffle pieces are written and read on the same disk with no network between them.

Checking from inside the program

A running application can tell you which arrangement it got. The property spark.submit.deployMode holds client or cluster, and spark.master holds the master URL.

conf = spark.sparkContext.getConf()
print(conf.get("spark.submit.deployMode"), conf.get("spark.master"))

Do

Run scheduled production jobs in cluster mode, so a launcher reboot can't end them
Use client mode for shells and notebooks, where results must reach your screen
Launch client-mode work from a machine on the cluster's network, close to the executors
Look for a cluster-mode job's output in the driver log on its node

Avoid

Don't collect() a large result to a laptop driver; write it to storage instead
Don't run nightly jobs in client mode from a laptop
Don't expect a cluster-mode driver's print() to reach your terminal

A nightly job ends with print(report.collect()) and is launched with --deploy-mode cluster from a scheduler machine. Where do the rows end up?

In the scheduler machine's memory, printed to the terminal that launched the job
Spread across the executors' logs, with each task printing its own piece
In the driver's memory on a cluster node, printed into that driver's log
Nowhere, because a cluster-mode driver refuses to run collect() at all

About this lesson

This section is part of the Execution Model: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.