The 60-Second Walk-Through

Concepts covered: Spark Execution Model, Distributed Primitives

Here is the answer itself, the version you should be able to give in about a minute with no notes. There's exactly 1 driver process. It runs your program, turns your code into a plan, cuts that plan into stages and tasks, and decides which executor gets which task. It does no row-level processing itself; it only receives what an action like collect() returns. There are many executor processes, usually spread across separate machines, and they do all the row-level work, from the read through the aggregation. A cluster manager owns the pool of machines and hands executors to your application when it starts. That is usually YARN or Kubernetes, though Spark also ships its own standalone manager.

Then the unit of work. Your data is split into partitions, and 1 task processes 1 partition. The driver groups the plan into stages, and a new stage begins wherever rows have to move between executors so that rows sharing a key end up together, which is a shuffle. Inside a stage, each task works through its own partition with no network movement at all. Finish by naming the trigger: nothing runs until an action like count() or collect() or a write, at which point the driver ships the tasks out, the executors run them, and the driver gathers the result. That is the answer, end to end, and at that point you stop.

Watch the path once, then narrate it

The program below reads an orders table as 3 partitions and keeps the rows over 50, then groups by region and collects. Watch where each of the walk-through's beats happens. The filter runs inside the first stage, each of the 3 tasks dropping rows from its own partition without talking to any other. Each of those tasks also pre-sums its surviving rows per region, so the groupBy shuffle sends partial sums, not raw rows, to the task that owns each region, and the second stage adds them up. The collect() is the action; until that line, Spark has only recorded the plan.

result = (orders
    .filter(F.col("amount") > 50)
    .groupBy("region")
    .agg(F.sum("amount").alias("revenue"))
    .collect())

That run is the whole model in 8 rows: 2 stages, because there is 1 shuffle; 3 tasks in the first stage, because the table was read as 3 partitions; 3 tasks in the second, because that is the shuffle partition count the program was given; and 1 driver that did no row-level work and collected 3 result rows at the end. In the room you won't have the picture, so narrate it the way the animation plays it, left to right, 1 stage at a time, and you'll never lose your place.

The relationship most answers blur

The relationship to land hardest, because so many answers blur it, is partition to task. They're different things. A partition is a chunk of your data sitting in a file or in executor memory. A task is the unit of work that processes 1 of those chunks, and it's what a core executes. The 1-to-1 mapping is where Spark's parallelism comes from: 1,000 partitions means 1,000 tasks, and a cluster with 100 cores runs them 100 at a time in 10 waves. If you can say "1 task processes 1 partition, so the partition count is roughly how parallel the job can get," you have shown the model in a single sentence.

The partition count on the far side of a shuffle is set by configuration rather than by your data, and naming that is worth doing. spark.sql.shuffle.partitions sets how many partitions the stage after a groupBy or a join starts from, whatever the input looked like. With adaptive query execution on, the default since Spark 3.2, Spark may coalesce them into fewer tasks, so the setting is the upper bound you start from. The interviewer hears that and knows you have looked at a stage with far more tasks than the data deserved, which is a lived detail, and it costs you 1 sentence.

• 200

Make the trigger concrete

If the interviewer looks like they want a touch more, make the lazy trigger concrete by contrasting 2 lines. You write a filter and a groupBy and nothing happens: Spark records the plan and returns to you instantly. You write count(), and only now does the whole chain fire, tasks ship to executors, and a number comes back. Calling the action the trigger, rather than letting it sound like another item in the list, is the detail that marks a real mental model over a memorized order, and it sets up the shuffle follow-up you will get next.

df = spark.read.parquet("s3://shop/orders/")   # nothing runs
big = df.filter(F.col("amount") > 50)           # nothing runs
by_region = big.groupBy("region").count()       # nothing runs
by_region.collect()                             # now: 1 job, 2 stages, tasks ship

About this lesson

This section is part of the The Architecture Recital lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.