How a Spark Job Runs

You already know what a query is. You write SELECT, the database figures out how to run it, and rows come back. Spark looks like that on the surface: you write a transformation, you get a DataFrame, you ask for results. But there is no single database under it. There is a fleet of machines, and your one line of code becomes hundreds of small jobs that run in parallel across them. This beginner tier builds the mental model of that machine, from the cluster down to a single task.

The Cluster: Who Plans, Who Works

Daily Life

Interviews

DriverExecutorCluster Manager

Driver

The Brain

Runs your code, builds the plan, and hands out work. There is exactly one driver per job. It does not process data itself; it directs.

Executor

The Hands

A process on a worker machine that actually reads, filters, and aggregates data. A cluster has many. They do all the real lifting.

Cluster Manager

The Landlord

Owns the machines and rents out CPU and memory. When your job starts, it asks the manager for executors and gets some.

When you run a query against Postgres, one process does the work. Spark splits that single role into three. The driver is the process running your program: it holds your code, builds the execution plan, and decides what work needs doing. It is the only part that sees the whole job. The executors are separate processes on separate machines that do the actual data crunching. The cluster manager is the layer that owns the pool of machines and hands executors to your job when it asks.

The mental shift from a single database

The hard part of this for a SQL person is that the driver does not touch your data. It never sees the rows. It builds a plan, ships that plan to the executors, and waits. If you accidentally pull all your data back to the driver (we will see how), you have collapsed a distributed system back into one machine, and it will fall over. The whole point of Spark is that the data stays spread across the executors and the work goes to the data, not the other way around.

✓Best Practice

✗Avoid

TIP

Partitions: The Unit of Parallelism

Daily Life

Interviews

Your data does not arrive at an executor as one big table. Spark splits it into chunks called partitions. A partition is a contiguous slice of the rows, typically targeted around 128 MB, that lives in memory on one executor. A billion-row table might be 8,000 partitions. This split is the single most important idea in Spark, because it is the unit of parallelism: one task processes exactly one partition.

Why this is the number that matters

Because tasks map one-to-one onto partitions, the partition count is the parallelism ceiling. If your data is in 4 partitions, at most 4 things can happen at once, no matter how many machines you rented. If it is in 8,000 partitions and you have 200 CPU cores, Spark works through them 200 at a time in waves. A SQL engine hides this from you; Spark makes it your decision, because getting it wrong is one of the most common ways a job runs slowly.

Partition count	What happens	Symptom
Far too few (e.g. 4)	Most of the cluster sits idle; a few huge tasks	Slow job, low CPU usage, possible memory blowups
Far too many (e.g. 500,000 tiny ones)	Scheduling overhead per task dominates the real work	Slow job, driver busy, tiny task times
About right (~128 MB each)	Every core stays busy, tasks finish in seconds	Steady high CPU usage across the cluster

The 128 MB target is a default, not a law: it is set by spark.sql.files.maxPartitionBytes and it exists because that size balances task startup cost against keeping a partition in memory. You will tune it later. For now, hold onto the shape: a partition is a chunk of rows, and one task chews through exactly one chunk.

Transformations vs Actions

Daily Life

Interviews

Here is the thing that surprises everyone coming from SQL. When you write df.filter(...).groupBy(...).agg(...), nothing runs. Spark does not read a single row. You have only described work. The description is called a transformation, and transformations are lazy: they build up a plan and return immediately. The data moves only when you call an action.

✓Best Practice

✗Avoid

Why a database can hide this and Spark cannot

In SQL, a statement is a complete unit: you send it, it runs, you get rows. There is no gap between describing and executing. Spark deliberately keeps that gap open so it can see your entire chain of transformations before it commits to a plan. Seeing the whole chain lets it collapse steps, push filters down to the read, and skip work you do not actually need. Laziness is not a quirk; it is how Spark optimizes. The cost is that a stack trace points at an action like count(), even though the real culprit is a transformation thirty lines earlier.

TIP

Cores and Slots

Daily Life

Interviews

An executor is not a single worker. It has N cores, and each core can run one task at a time. So an executor with 5 cores is processing 5 partitions simultaneously. Think of a core as a slot: a place a task can be running right now. Your total parallelism is the sum of all slots across all executors.

1	total slots = executors x cores - per - executor 10 executors x 5 cores = 50 slots -> 50 tasks running AT the same TIME -> 50 partitions processed per wave

Slots and partitions decide your wall-clock time

Put the last two sections together. You have a fixed number of partitions (the work) and a fixed number of slots (the workers). If you have 200 partitions and 50 slots, Spark runs them in 4 waves: 50, then 50, then 50, then 50. If one partition is much bigger than the rest, its task runs long and a slot is stuck on it while others sit idle. That is the seed of every skew problem you will ever debug, and it falls straight out of the slots-and-partitions arithmetic.

This is the level at which 'add more executors' is sometimes right and sometimes useless. More slots help only if you have enough partitions to fill them. Adding 100 slots to a 4-partition job does nothing: 96 slots sit empty. The arithmetic tells you instantly whether more hardware can possibly help.

A Job's Life, End to End

Daily Life

Interviews

Now we narrate one full run, using only the pieces from B1 through B4. This is the answer to the single most common Spark interview opener: walk me through how Spark runs a job. The trick is to answer along the path the work actually travels, not as a list of vocabulary.

You call an action (say count()). Until now everything was lazy; this is what wakes the job up.

The driver turns your chain of transformations into a plan and figures out which data to read and how to split it into partitions.

The driver asks the cluster manager for executors, and packages the work into tasks, one task per partition.

The driver ships those tasks to executor slots. Each slot runs its task against its one partition.

Executors send results (or a summary, like a partial count) back to the driver, which assembles the final answer and hands it to you.

The one-sentence version you should be able to say cold

An action triggers the driver to plan the work, split the data into partitions, ask the cluster manager for executors, send one task per partition to the executor slots, and collect the results back. If you can say that smoothly and then stop, you have answered the question better than most candidates, who either recite API names or never reach the word 'partition.'

KEY TAKEAWAYS

Driver plans and directs; executors do the data work; the cluster manager owns the machines.

Data is split into partitions; one task processes one partition.

Transformations are lazy descriptions; an action is what makes the job actually run.

Parallelism is total slots (executors x cores); wall-clock time is roughly partitions / slots in waves.

The execution path is: action -> plan -> partitions -> tasks -> slots -> results.

Your query is a promise. Something has to keep it.

Category: SPARK
Difficulty: beginner
Duration: 12 minutes
Challenges: 7 hands-on challenges

Topics covered: The Cluster: Who Plans, Who Works, Partitions: The Unit of Parallelism, Transformations vs Actions, Cores and Slots, A Job's Life, End to End

Lesson Sections

The Cluster: Who Plans, Who Works
When you run a query against Postgres, one process does the work. Spark splits that single role into three. The driver is the process running your program: it holds your code, builds the execution plan, and decides what work needs doing. It is the only part that sees the whole job. The executors are separate processes on separate machines that do the actual data crunching. The cluster manager is the layer that owns the pool of machines and hands executors to your job when it asks. The mental shi
Partitions: The Unit of Parallelism
Your data does not arrive at an executor as one big table. Spark splits it into chunks called partitions. A partition is a contiguous slice of the rows, typically targeted around 128 MB, that lives in memory on one executor. A billion-row table might be 8,000 partitions. This split is the single most important idea in Spark, because it is the unit of parallelism: one task processes exactly one partition. 1 : 1 Why this is the number that matters Because tasks map one-to-one onto partitions, the
Transformations vs Actions
Here is the thing that surprises everyone coming from SQL. When you write df.filter(...).groupBy(...).agg(...), nothing runs. Spark does not read a single row. You have only described work. The description is called a transformation, and transformations are lazy: they build up a plan and return immediately. The data moves only when you call an action. Why a database can hide this and Spark cannot In SQL, a statement is a complete unit: you send it, it runs, you get rows. There is no gap between
Cores and Slots
An executor is not a single worker. It has N cores, and each core can run one task at a time. So an executor with 5 cores is processing 5 partitions simultaneously. Think of a core as a slot: a place a task can be running right now. Your total parallelism is the sum of all slots across all executors. Slots and partitions decide your wall-clock time Put the last two sections together. You have a fixed number of partitions (the work) and a fixed number of slots (the workers). If you have 200 parti
A Job's Life, End to End
Now we narrate one full run, using only the pieces from B1 through B4. This is the answer to the single most common Spark interview opener: walk me through how Spark runs a job. The trick is to answer along the path the work actually travels, not as a list of vocabulary. The one-sentence version you should be able to say cold An action triggers the driver to plan the work, split the data into partitions, ask the cluster manager for executors, send one task per partition to the executor slots, an