Explain Spark Architecture

Concepts covered: Batch Processing, Partitioning Strategies

The architecture question arrives as 'walk me through what happens when I submit a Spark job', and the interviewer scores it on whether you can name each component, say what it holds, and trace a transformation from the line of code to the tasks that run it. A complete answer draws 3 kinds of process and then shows how a job becomes stages and stages become tasks. Start with the processes.

The driver is the JVM that runs your main. It holds the SparkSession, turns your transformations into a logical plan, optimizes that plan, cuts it into stages at every shuffle, and hands individual tasks to executors. It also keeps the bookkeeping: which task ran where, which shuffle block lives on which executor, which partitions are cached. Executors are JVMs the cluster manager launches on worker nodes. Each has a fixed core count and a fixed heap, runs 1 task per core at a time, caches partitions in its memory, and serves the shuffle files it wrote to any executor that asks. In most deployments the cluster manager is YARN or Kubernetes, and its only job is to allocate those containers and tear them down; it never sees your data or your plan. A 2 TB job and a 2 MB job look identical to it.

From one action to thousands of tasks

from pyspark.sql import functions as F

events = spark.read.parquet('/lake/events/dt=2026-10-22')
paid = events.filter(events.plan == 'premium')
per_title = paid.groupBy('title_id').agg(F.sum('seconds').alias('watched'))
per_title.write.parquet('/lake/watch_per_title/dt=2026-10-22')

Nothing runs until the write on the last line, because every transformation is lazy: read only extends the plan, and so do filter and groupBy. The write is an action, and 1 action produces 1 job. The driver then walks the plan backwards looking for shuffles. filter is narrow: each output partition depends on exactly 1 input partition, so it stays in the same stage as the read. groupBy is wide: the rows for a given title_id are spread across every input partition and must be moved to one place, so the driver cuts a stage boundary there. The job has 2 stages. Stage 1 reads and filters, with 1 task per input split; at the default spark.sql.files.maxPartitionBytes of 128 MB, 1 TB of Parquet becomes about 8,192 tasks. Stage 2 sums per title, with 1 task per shuffle partition; at the default spark.sql.shuffle.partitions of 200 it runs 200 tasks however much data arrived.

Why the shuffle is the unit of cost

At the stage boundary every stage 1 task hashes each row's title_id into 1 of 200 buckets, sorts its output by bucket, and writes 1 file with an index marking where each bucket begins to its executor's local disk. Every stage 2 task then opens a connection to each executor that wrote a segment for its bucket and pulls those bytes across the network. With 8,192 upstream tasks and 200 downstream tasks that is up to 1,638,400 fetches, and each one costs a disk read and a network transfer before the bytes are deserialized. The filter before the shuffle is the reason the job is affordable: if premium viewers produce 10% of the rows, the shuffle moves 100 GB instead of 1 TB. Each question later in this lesson comes down to how many bytes cross that boundary and how evenly they spread over the 200 tasks.

About this lesson

This section is part of the The Spark Deep Dive: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.