IntermediateSpark · 15 min · 4 challenges
The Architecture Recital
Priya is 40 minutes into a data engineering loop at Uber, a company that has written publicly about building a shuffle service for Spark. The interviewer opens with a familiar prompt: "Walk me through what happens when I run a groupBy and a join over a month of trip data." Priya lists the driver, the cluster manager, executors, stages and tasks, in the right order, in under a minute. The interviewer nods and asks, "Which of those steps would you expect to dominate the runtime, and by how much?" Priya has the names but no numbers, and the conversation stalls. The opening prompt was a depth probe: the answer it wanted pairs each stage with what it costs and a figure to size it. This lesson rebuilds that answer around the shuffle, the 3 defaults that size a job, and the follow-up about why moving rows between executors hurts.
What you will be able to do
Recognize the architecture walk-through as a depth probe in any of its phrasings, and know what the interviewer is listening for.
Structure the answer by stage, placing each stage boundary where a shuffle Exchange sits in the plan.
Attach a cost and a number to each stage using the 128MB split, the 200 shuffle partitions and the 10MB broadcast threshold.
Spot a name-list answer in your own practice and repair it with a location, a size and a check.
Answer "why are shuffles expensive?" with the write, the barrier, the fetch, the merge and skew, naming the Spark UI metric for each.
The walk-through question shows up in Spark interviews under many phrasings, and by now the interviewer assumes you can name the driver, the executors and the cluster manager. The question underneath is about weight: which steps move data, how many bytes, and what that movement does to the wall clock. A list of components answers none of that. The answers that pass this probe attach a cost to each step before the interviewer has to ask for it.
One question, 5 phrasings
| How it is phrased | What it probes | The figure that answers it |
|---|---|---|
| "What happens, step by step, when this job runs?" | Whether you can find the shuffle boundaries in the code | Stages in the plan and tasks per stage |
| "What is the difference between a transformation and an action?" | Whether you know nothing runs until the action, and what the action sets off | Shuffles in the plan the action triggers |
| "This job got slower when the input doubled. Where would you look?" | Whether you know which stage scales with data volume | Shuffle Write per stage |
| "Explain the driver and the executors." | Whether you know what lives where and what crosses the network | Bytes collected to the driver, bytes moved between executors |
| "Why does this join take 40 minutes?" | Whether you check the join strategy before tuning anything else | The smaller side's size against the 10MB threshold |
The way to recognize the probe is the follow-up. If the first answer lists components, the next question is "which of those is expensive?" or "how many tasks would that run?", and you end up producing under pressure what you could have volunteered. If the first answer already places the shuffles and sizes them, the follow-ups become a discussion of tradeoffs, which is where you want the conversation to be.
What the probe listens for
✓A correct walk-through
- The driver turns the code into a plan when an action runs
- The plan splits into stages at each shuffle
- Each stage runs a task per partition
- Executors run the tasks and keep shuffle files on local disk
✗A walk-through with weight
- Which line of code causes each shuffle, and which stage moves the most bytes
- How many tasks run, from file size and the 200 shuffle default
- Whether the join broadcasts or shuffles both inputs, and the size that decided it
- Which Spark UI metric would confirm or refute each guess
Both columns describe the same machinery. The right one adds 3 things to every claim: a location (the line of code that causes the movement), a size (bytes and task counts, rough ones included) and a check (where in the Spark UI the guess gets confirmed). An answer carrying all 3 lets the interviewer push on any part of it and find something solid underneath.
Before you start talking
- ▸Where in this code does data cross the network?
- ▸How many partitions does each stage run with?
- ▸Which number would I quote, and where would I read it in the UI?
A worked opener
Suppose the interviewer pastes this job and asks what happens when it runs. The trips table holds March, 500GB of Parquet. The cities table holds a few thousand cities with their boundary polygons, about 6MB on disk.
Count the data movement before naming anything. At about 6MB, cities sits under the 10MB spark.sql.autoBroadcastJoinThreshold, so Spark broadcasts it: the driver collects the table and a copy reaches every executor, while trips never moves for the join. The groupBy adds a hash shuffle on city_name. The orderBy adds a range shuffle on revenue. Two shuffles split the write into 3 stages.
| Stage | What runs in it | Tasks |
|---|---|---|
| 1 | Scan trips, filter, broadcast hash join, partial sum per city | About 4000, a task per 128MB split |
| 2 | Final sum per city, then the write for the range shuffle | 200 planned by the shuffle default. Adaptive execution, on by default since Spark 3.2, coalesces these tiny partitions, so the UI shows far fewer tasks |
| 3 | Sort by revenue, write the report | Up to 200 planned, a task per range, coalesced the same way |
Said aloud, that becomes the opener: "Nothing runs until the write. The driver plans this as a broadcast join and 2 shuffle exchanges, so 3 stages. Stage 1 is the expensive one, about 4000 tasks reading 500GB, but partial aggregation means each task ships at most a row per city into the shuffle. Before tuning anything I'd confirm the broadcast in the SQL / DataFrame tab." Each sentence carries a count or a decision, so the interviewer can follow up on any of them and get a real answer.
The 6MB figure is Spark's size estimate from file statistics. A filter or an explode on the small side can make the real size differ from the estimate, which is another reason the plan check comes before any tuning.
TIP
When the interviewer shows code, count the shuffles out loud before naming a single component. It shows you read plans, and it gives every later sentence a place to hang its number.
An answer ordered by component (driver, then cluster manager, then executors) describes the org chart of a Spark application. An answer ordered by stage describes the job's timeline, and the timeline is where the cost lives, because every stage boundary is a point where data is written, moved and read again. Reordering the same facts around stages is the change the rest of this lesson builds on.
Component order against stage order
✓Ordered by component
- The driver creates a SparkSession and requests resources
- The cluster manager launches executors
- The DAG scheduler splits the work into stages
- Executors run tasks and return results
✗Ordered by stage
- Plan: the write triggers a physical plan with a broadcast join and 2 shuffles
- Stage 1: about 4000 scan tasks filter, join and pre-sum, then write shuffle files
- Stage 2: 200 reducers fetch those files, finish the sum, write range output
- Stage 3: sorted ranges are written out as the report
The stage-ordered version still mentions the driver and the executors, but each appears at the moment it does work, attached to a count. The interviewer can interrupt at any stage and ask "why that many?", and the answer is already in the sentence.
500GB Parquet
6MB
broadcast
scan
filterjoinpartial sumPARTITIONS4000
executor local disk
final sum per city
PARTITIONS200
executor local disk
sort by revenue
writePARTITIONS200
Parquet output
The March job laid out by stage. Each storage box is a shuffle boundary: tasks on its left write grouped files to local disk, and tasks on its right fetch them over the network.
Where the boundaries fall
Spark cuts a stage at every shuffle Exchange in the physical plan. Operators between exchanges (scan, filter, project, the map side of an aggregate, a broadcast hash join) run pipelined inside the same task over the same partition, with no intermediate result written anywhere. That pipelining is why a filter adds almost no cost: it rides inside a task that is already reading the data. An Exchange is where a task stops, writes its output grouped by destination partition, and hands off to tasks in the next stage that start by fetching it.
| Operation | Plan operators | New stage? |
|---|---|---|
| filter, select, withColumn | Filter, Project | No |
| groupBy then agg | HashAggregate around Exchange hashpartitioning | Yes, a hash shuffle on the keys |
| join, small side under 10MB | BroadcastExchange, BroadcastHashJoin | No, the large side stays in place |
| join, both sides over 10MB | Exchange under each side, SortMergeJoin | Yes, both inputs shuffle |
| orderBy | Exchange rangepartitioning, Sort | Yes, a range shuffle |
| groupBy after repartition on the same key | HashAggregate reusing the partitioning | No second shuffle |
In the SQL / DataFrame tab, pipelined operators sit inside boxes labelled WholeStageCodegen with a number, and the Exchange nodes sit between those boxes, so the stage boundaries are visible in the graph.
Watch a boundary form
This run filters and groups 9 trips read as 3 partitions, with 3 reducers on the other side. Watch the first stage do the scan, the filter and a partial sum inside each task, then watch the partial sums cross to the reducers.
city_totals = (trips.filter("fare > 5").groupBy("city").sum("fare"))city_totals.collect()
Preparing the animation
Two trips fail the filter (a 4.0 fare in SF and a 3.5 fare in LA) and never leave their tasks. Each map task sends at most 1 partial sum per city across the boundary, so this shuffle carries at most 9 rows however many trips each partition held. The driver collects 3 totals: SF 39.5, NYC 34.5 and LA 22.0. The same property explains stage 1 of the March job: heavy on reading, light on shuffling, because each task holds far more trips than there are cities.
The spoken shape
Four beats, a number in each
- Planwhat the action triggersThe write or collect makes the driver build the physical plan. Name its shuffle exchanges and the join strategy it chose.
- Stagesa stage per stretch between exchangesCount them out loud as shuffles plus 1. Say which stage reads the most bytes and which writes the most.
- Taskspartitions decide parallelismInput splits of 128MB set the scan's task count. The 200 shuffle partitions are the planned count after each exchange, which adaptive execution may merge when they turn out small. Divide by cores to get waves.
- Shuffle and resultwhat crosses the networkSay roughly how many bytes cross each boundary, and whether the result returns to the driver or lands in storage.
TIP
Budget about 90 seconds for the first pass. A beat with no number in it is a candidate to cut, since the interviewer will ask for that number next.
An interviewer asking for a number wants an estimate built from defaults you can defend, with the arithmetic visible so they can correct an input instead of rejecting the answer. Three settings supply most of those numbers, and all 3 are defaults you can quote without a cluster in front of you.
The 3 defaults that size a job
| Setting | Default | What it sizes | Cost when it is wrong |
|---|---|---|---|
| spark.sql.files.maxPartitionBytes | 128MB | Tasks in each file scan | Too small: thousands of short tasks and scheduling overhead. Too large: fewer, longer tasks and more memory per task |
| spark.sql.shuffle.partitions | 200 | Tasks in every stage after a shuffle | Too few: gigabytes per task and spill. Too many: millions of tiny shuffle blocks |
| spark.sql.autoBroadcastJoinThreshold | 10MB | Whether a join shuffles at all | Below it: the small side is copied to every executor. Above it: both sides shuffle and sort |
Each default converts a mechanism into a count. File size over 128MB gives the scan's tasks. The shuffle default is the number of partitions Spark plans after every shuffle, whatever the volume. With adaptive execution enabled, the default since Spark 3.2, Spark can merge small partitions down from that number once the map stage has run, but coalescing never splits a large one: a 2GB reducer for a window or an aggregate stays 2GB. That makes 200 the default to question first on a large job. The broadcast threshold decides what is often the largest single cost in a join: whether the big side moves at all.
One job, worked end to end
Change the March job so it numbers each rider's trips in pickup order, a window partitioned by rider_id. A window gets no partial aggregation, so every row that survives the scan crosses the shuffle. The cluster runs 50 executors with 4 cores each, and a sample run shows about 400GB of Shuffle Write on the map stage.
| Quantity | Arithmetic | Result |
|---|---|---|
| Scan tasks | 500GB / 128MB | 4000 |
| Task slots | 50 executors x 4 cores | 200 |
| Scan waves | 4000 / 200 | 20 |
| Data per reducer at 200 | 400GB / 200 | 2GB |
| Data per reducer at 2000 | 400GB / 2000 | about 200MB |
| Shuffle blocks at 2000 | 4000 x 2000 | 8,000,000 |
| Average block at 2000 | 400GB / 8,000,000 | about 52KB |
Read the table as 2 problems trading against each other. At the default, each reducer sorts 2GB for the window, which will likely spill to disk unless every task has well over 2GB of execution memory. Raising spark.sql.shuffle.partitions to 2000 brings each reducer near 200MB, a size that usually sorts in memory, but multiplies the shuffle into 8 million blocks averaging 52KB, and many small remote reads move fewer bytes per second than a few large ones. A common target is 100MB to 200MB per shuffle partition, and the interviewer wants to hear that both ends of the dial cost something.
Sentences that carry their arithmetic
- ▸500GB at 128MB a split is about 4000 tasks; on 200 cores that is 20 waves.
- ▸400GB over 200 reducers is 2GB a task, which will spill during the sort.
- ▸2000 partitions gets that near 200MB, at the price of 8 million shuffle blocks.
- ▸Cities is about 6MB, under the 10MB threshold, so the join broadcasts.
The join decision outweighs the rest
Go back to the original March job and its 6MB cities table. Whether that join broadcasts is the first number to state and the first to verify, because the 2 strategies move data at different scales. A broadcast costs driver memory to collect the small side, then a 6MB copy to each of the 50 executors. A sort-merge join shuffles both inputs and sorts both, and the large side dominates: every completed trip from the 500GB scan is hashed on city_id and crosses the network, hundreds of GB against the broadcast's megabytes.
300MB
total broadcast traffic for cities
0 bytes
of trips moved for the join
Set those 2 figures beside the window job's 400GB shuffle: a sort-merge join on the March trips would add a shuffle on that scale, while the broadcast stays in megabytes. That sets the order of the answer: confirm the join strategy, then size the shuffle partitions, then think about scan waves.
TIP
Say the arithmetic along with the result. If the interviewer says the cluster has 400 cores, you redo a division out loud and the estimate survives instead of collapsing.
A common way to fail this probe is with an answer that is complete and correct. It names every component, puts them in startup order and contains no mistake to point at. What it lacks is any statement that could turn out wrong: no location, no count, no choice. With nothing falsifiable on the table, the interviewer pushes for numbers, and you end up producing them under questioning instead of leading with them.
The answer that stalls
A recital of names
When the job is submitted, the driver starts and creates a SparkSession. It asks the cluster manager for resources, and the cluster manager launches executors. The driver builds a DAG of the transformations, the DAG scheduler splits it into stages, and the task scheduler sends tasks to the executors. The executors run the tasks, and the results come back to the driver.
Every sentence there is accurate, and it fits any Spark job ever written, which is the problem. It never touches the code on the screen, so the interviewer learns nothing about whether you can read this job's plan. The next few minutes go to questions you could have pre-empted: how many stages, which one is slow, does the join shuffle. Each of those costs time the round could have spent on tradeoffs.
It would work unchanged for a different job.
No sentence in it could be disproved by the Spark UI.
The cluster manager gets as many words as the shuffle.
It ends when the results reach the driver, with no cost named on the way.
Three gaps, three fixes
Hold the recital against the March job and its gaps line up with the location, size and check that the depth probe listens for.
| Gap | What the recital says | What it should say |
|---|---|---|
| Location | "splits it into stages" | "the join broadcasts at 6MB; the groupBy and the orderBy each add a shuffle, so 3 stages" |
| Size | "sends tasks to the executors" | "about 4000 scan tasks, 200 planned for each later stage, 20 waves on 200 cores" |
| Check | nothing | "I'd confirm BroadcastHashJoin and the 2 exchanges in the SQL / DataFrame tab" |
The repaired answer
The same facts, with weight
Nothing runs until the write. The driver plans it as 3 stages: the join broadcasts since cities is about 6MB, and the groupBy and orderBy each add a shuffle. Stage 1 carries the cost, about 4000 tasks over 500GB in 20 waves on 200 cores, and partial aggregation keeps its shuffle output to a row per city per task. Stages 2 and 3 work on small data. If it ran slow, I'd open the SQL / DataFrame tab to confirm the broadcast, then compare Shuffle Write across the stages.
It runs about 20 seconds longer than the recital and names fewer components. The driver appears once, as the thing that plans. The cluster manager never appears, because nothing in this question turns on it. Leaving a name out is a choice the interviewer can see, and it reads as knowing which parts carry the cost.
Repair habits
✓Do
- Open by counting the exchanges in the code on the screen
- Give every stage a task count, even a rough one
- Name the join strategy and the size that decided it
- Close with the Spark UI metric you would check first
✗Don't
- List components in startup order with no tie to the code
- Spend a sentence each on the DAG and task schedulers unprompted
- Say "it depends" without naming what it depends on
- Quote a number you could not derive if asked
Try it
Rewrite this recital sentence for the window job from the previous section, adding a location, a size and a check: "Spark shuffles the data so each partition has the right keys, then runs the window on each partition." Write your version before reading the one below.
One version: "The window partitions by rider_id, so every scanned row crosses a hash shuffle, about 400GB from 4000 map tasks. At the 200 default that is 2GB per reducer, so I'd raise shuffle partitions toward 2000 and check Spill (Disk) on the window stage."
TIP
Practice the repair on code you did not write. Take any job from your team's repository, count its exchanges, size its scan, and say the 3-sentence version out loud in under a minute.
Once the walk-through has placed the shuffles, the next question is close to predictable: "why are shuffles so expensive?" The thin answer is "because data goes over the network", which names 1 cost out of several and gives no way to tell which dominates. The answer the interviewer wants follows a shuffled byte from the task that produces it to the task that consumes it, names the cost at each step, and says where in the Spark UI each cost shows up.
Follow a byte through the shuffle
| Step | What it costs | Where the Spark UI shows it |
|---|---|---|
| Map-side write | Each row is serialized, grouped by destination partition, compressed with LZ4 and written to local disk with an index file | Shuffle Write Size / Records on the map stage |
| Barrier | The reduce stage starts only after every map task has finished, so the slowest map task sets the start time | The stage page's Event Timeline, a long last task |
| Fetch | Each reducer pulls its block from every map output over the network, up to map tasks times reducers in total | Shuffle Read Blocked Time, Shuffle Remote Reads |
| Merge | Blocks are decompressed, deserialized, then sorted or hashed; whatever does not fit in memory spills | Spill (Memory), Spill (Disk) |
| Skew | A hot key sends its whole volume to a single reducer | Max far above Median in Summary Metrics |
| Lost output | Shuffle files live on the machine that wrote them; losing them reruns the map tasks that produced them | FetchFailed in a task error, the map stage listed again as a retry |
The costs fall into 2 kinds. Writing, fetching and merging scale with bytes, so they dominate a shuffle like the window job's 400GB. The barrier and skew scale with the slowest task, so they dominate whenever a partition is far larger than the rest, whatever the total. Saying which kind you expect for the job in front of you is the second sentence of a good answer.
Put figures on it
The window job gives every cost a figure. The map stage writes about 400GB of compressed shuffle files to local disk, and the reduce stage fetches the same volume back over the network. At 2000 reducers that fetch is 8 million blocks of about 52KB, each a separate request. The reduce stage waits for all 4000 map tasks, so a single task stuck on a slow file leaves most of the 200 cores idle at the tail of the stage. If a fleet account has millions of trips, the reducer holding that rider_id sorts them alone while the rest finish.
The 30-second answer
Why shuffles cost what they cost
A shuffle splits the work into a writing stage and a reading stage with a hard barrier between them. On the map side, every task serializes its output, groups it by destination partition, compresses it and writes it to local disk. On the reduce side, every task fetches its piece from every map task over the network, so the block count is map tasks times reducers, then deserializes the rows and sorts or hashes them, spilling whatever does not fit. The reduce stage cannot start until the slowest map task finishes, and a hot key puts its whole volume on a single reducer. In the UI I'd read Shuffle Write per stage, Shuffle Read Blocked Time and Spill (Disk), then compare the max task to the median.
See both inputs move
This run joins 9 trips to 3 cities with the broadcast threshold set to -1, which forces a sort-merge join. Both tables are read as 3 partitions and both are hashed on city_id into 3 reducers, so every trip and every city row crosses the boundary before a single pair is matched.
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)joined = trips.join(cities, "city_id")joined.collect()
Preparing the animation
Delete the spark.conf.set line and the plan changes shape: cities, a few hundred bytes against a 10MB threshold, becomes a BroadcastExchange, and trips is joined inside its own scan tasks without crossing the network. That contrast is the strongest answer to "how would you cut shuffle cost?": the cheapest shuffle is the one the plan never contains.
When they push further
The usual next question is how you would reduce the cost. Each lever below removes bytes or a slow task from a row of the table above.
Filter and project before the shuffle so fewer bytes reach the map-side write.
Broadcast a side under 10MB, or add cities.hint("broadcast") when the size estimate is missing or wrong.
Favor aggregates that shrink on the map side, such as sum and count; collect_list ships every value.
If a join or repartition you needed already hash-partitioned the data on a key, a groupBy on that key adds no shuffle.
Size shuffle partitions toward 100MB to 200MB each instead of keeping the 200 default.
Salt a hot key across several reducers when its task's input dwarfs the median.
TIP
If the interviewer asks which shuffle cost matters most, answer for the job in front of you: bytes for a wide window or join, the slowest task for a skewed key. A ranking with no job attached sounds memorized.
❯❯❯PUTTING IT ALL TOGETHER
> A rides analytics team runs a nightly job that numbers every rider's March trips with a window on rider_id, joins the result to a 6MB cities dimension and aggregates revenue per city. Trip volume has grown: the scan now reads 500GB, Shuffle Write on the window stage reads about 400GB, and the job has gone from 25 minutes to 90. The on-call engineer has to explain the slowdown to the team before proposing a fix.
Reading the plan first shows a broadcast join and 2 shuffle Exchanges, so the engineer knows the window's hash shuffle and the revenue aggregate's shuffle are the only places rows cross the network.
Explaining by stage puts the 4000-task scan first, the window's reducers second and the small aggregate last, which is the order in which the cost falls.
The arithmetic locates the slowdown: 400GB over the default 200 reducers is 2GB per task, and Spill (Disk) on the window stage confirms the guess.
The join needs no change because cities stays under 10MB, and saying so out loud rules out a fix the team might otherwise try first.
The proposed fix, raising shuffle partitions toward 2000, brings its own cost in 8 million blocks of about 52KB, so the engineer states that tradeoff alongside the fix.
KEY TAKEAWAYS
The architecture walk-through is a depth probe: the interviewer is listening for where data moves and what that movement costs.
Order the answer by stage. Boundaries sit at shuffle Exchanges, and each boundary is a write, a barrier and a fetch.
Three defaults give you numbers anywhere: 128MB per input split, 200 shuffle partitions, a 10MB broadcast threshold.
Show the arithmetic so the interviewer can change an input and watch the estimate move with it.
A name-list answer is accurate and unfalsifiable; repair it with a location, a size and a check.
Shuffle cost follows either the bytes (write, fetch, merge) or the slowest task (barrier, skew); say which one this job has.
Turn the Spark architecture walk-through into an answer that pairs every stage with what it costs and a number.
- Category
- Spark
- Difficulty
- intermediate
- Duration
- 15 minutes
- Challenges
- 4 hands-on challenges
Topics covered: A Depth Probe, Answer in Stages, Mechanics to Cost, With a Number, Giving the Junior Answer, They Ask Why Shuffles Are Costly
Lesson Sections
- A Depth Probe (concepts: Spark Execution Model)
The walk-through question shows up in Spark interviews under many phrasings, and by now the interviewer assumes you can name the driver, the executors and the cluster manager. The question underneath is about weight: which steps move data, how many bytes, and what that movement does to the wall clock. A list of components answers none of that. The answers that pass this probe attach a cost to each step before the interviewer has to ask for it. One question, 5 phrasings The way to recognize the p
- Answer in Stages (concepts: Shuffle Optimization)
An answer ordered by component (driver, then cluster manager, then executors) describes the org chart of a Spark application. An answer ordered by stage describes the job's timeline, and the timeline is where the cost lives, because every stage boundary is a point where data is written, moved and read again. Reordering the same facts around stages is the change the rest of this lesson builds on. Component order against stage order The stage-ordered version still mentions the driver and the execu
- Mechanics to Cost, With a Number (concepts: Shuffle Optimization)
An interviewer asking for a number wants an estimate built from defaults you can defend, with the arithmetic visible so they can correct an input instead of rejecting the answer. Three settings supply most of those numbers, and all 3 are defaults you can quote without a cluster in front of you. The 3 defaults that size a job Each default converts a mechanism into a count. File size over 128MB gives the scan's tasks. The shuffle default is the number of partitions Spark plans after every shuffle,
- Giving the Junior Answer (concepts: Shuffle Optimization)
A common way to fail this probe is with an answer that is complete and correct. It names every component, puts them in startup order and contains no mistake to point at. What it lacks is any statement that could turn out wrong: no location, no count, no choice. With nothing falsifiable on the table, the interviewer pushes for numbers, and you end up producing them under questioning instead of leading with them. The answer that stalls Every sentence there is accurate, and it fits any Spark job ev
- They Ask Why Shuffles Are Costly (concepts: Shuffle Optimization)
Once the walk-through has placed the shuffles, the next question is close to predictable: "why are shuffles so expensive?" The thin answer is "because data goes over the network", which names 1 cost out of several and gives no way to tell which dominates. The answer the interviewer wants follows a shuffled byte from the task that produces it to the task that consumes it, names the cost at each step, and says where in the Spark UI each cost shows up. Follow a byte through the shuffle The costs fa