A Tour With No Failure Modes
The most common answer to 'walk me through Spark's execution' at this depth is a polished tour: every phase named in order, every component described accurately, and no case where any of it produces a bad plan. Interviewers hear it often enough that it signals preparation more than experience. A correct tour scores as the baseline, and the follow-up questions are built to get past it.
- Parser (transform, text or DataFrame to unresolved plan)
- Analyzer (transform, names resolved against the catalog)
- Logical optimizer (transform, rule batches)
- Physical planner (transform, join strategy, cost model)
- Adaptive re-plan (quality gate, between query stages)
- Codegen (transform, whole-stage Java code)
- Tasks (consumer, one per partition)
- Parser -> Analyzer
- Analyzer -> Logical optimizer
- Logical optimizer -> Physical planner
- Physical planner -> Adaptive re-plan
- Adaptive re-plan -> Codegen
- Codegen -> Tasks
The same tour, with the failure modes left in
| Phase | What the tour says | What it leaves out |
|---|---|---|
| Analyzer | Resolves tables and columns | Its failures are loud: an unresolved column fails before a single task runs |
| Logical optimizer | Pushes filters down, prunes columns | Stops at a Python UDF, so a filter on a UDF result reads every row group |
| Physical planner | Chooses the join by size | Sizes come from stale statistics or file listings; a selective filter looks as big as its input |
| Adaptive re-plan | Fixes the plan at runtime | Measures compressed bytes, and only after a stage completes |
| Codegen | Fuses operators into one function | Falls back past `spark.sql.codegen.maxFields` (100) or when the generated method grows too large; a Python UDF breaks the fused pipeline |
| Tasks | Run in parallel | A hot key turns the stage's tail serial |
What the follow-ups are looking for
After a tour, the interviewer's next questions are predictable because they all point at the gap it skipped: 'What if the filter is very selective?', 'What if one customer has half the orders?', 'What does AQE actually measure?' Each is an invitation to describe a failure mode. A recital that already contains a couple of them frees the interviewer to go deeper, into how you chose a fix and what it cost, which is the part of this question they learn the most from.
Rewriting a paragraph of the tour
Take the sentence about adaptive execution, the one most tours end on, and rewrite it so it carries its own limit. The rewrite adds 1 sentence.
- As toured
- AQE re-optimizes the plan at runtime using real statistics.
- It coalesces partitions, switches to broadcast joins and handles skew.
- With its edge
- AQE re-plans at shuffle boundaries from the sizes map tasks report.
- Those sizes are compressed bytes, so a stage that expands rows can be coalesced to too few tasks.
- Its skew handling covers joins, so a skewed groupBy still needs a salted key or a two-step aggregate.
A self-check after your recital
- Did I name a decision the driver made before reading any data?
- Did I say which estimate fed that decision and where the estimate came from?
- Did I give a case where that estimate is wrong, with a number attached?
- Did I name the Spark UI view that would show the miss?
- Did I stop before the interviewer had to interrupt?
Scoring yourself against those 5 questions after a practice answer catches the tour pattern quickly, because a tour passes the last check and fails the first 4. Recording a 2-minute answer and reading it back as a transcript makes the gap visible: the transcript will be full of component names and short on numbers.
Scoring a sample tour
| Check | Result | Why |
|---|---|---|
| A decision made before data | Pass, barely | It names the join choice, but not that the driver fixed it |
| The estimate behind it | Fail | 'Both tables are large' hides that the filtered side was assumed to be 2GB |
| A wrong case with a number | Fail | Nothing says the filter keeps 8MB, so the lost broadcast never comes up |
| The UI view | Fail | No mention of `explain(mode="cost")` or the exchange sizes on the SQL tab |
| Stopped in time | Pass | About 25 seconds spoken |
One added sentence repairs all 3 failing checks: 'The planner assumed the purchase filter kept all 2GB of events, the job measures about 8MB, and I would confirm it by comparing explain(mode="cost") with the output rows on the SQL tab.' The tour keeps its order and its length stays under a minute; it gains the evidence an interviewer would otherwise spend several questions extracting.
A tour with no failure modes still fits a 60-second opener. The miss is offering it as the whole answer when the question asked how the system behaves on a particular query.
About this lesson
This section is part of the The Architecture Recital lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.