BeginnerSpark · 15 min · 3 challenges

The War Story

Priya sits down for her Amazon data engineer loop. Amazon tells candidates its interviews lean on behavioral questions tied to its Leadership Principles, so she expects one, and it comes: 'Tell me about a time you made a slow pipeline faster.' She has a real story. Her nightly Spark job missed its dashboard deadline every morning until she found a single config line forcing a sort-merge join against a tiny table. But she opens with the company, the team, the stack and the history of the job. Four minutes in, the interviewer cuts in: 'So what did you change?' The fix was good. The telling buried it. This lesson builds the telling: one story anchored on one number, the action reached in under 30 seconds, a measured before and after, a tour of the system, and the mechanism behind the fix when the follow-up comes.

What you will be able to do

Pick a war story whose root cause shows up in the Spark plan, and open with a headline sentence carrying its number.

Keep context to 2 sentences and reach your own action within 30 seconds, naming what each Spark UI tab showed you.

Present a before and after in runtime, shuffle exchanges, shuffle write and spill, plus a query showing the output did not change.

Give a 30-second tour of the system with a size on every input, ending on the box that shuffled.

Answer 'why did that work?' by tracing where the data moved before, where it moves now, and what each movement cost.

One Story, One Number

Daily Life

Interviews

One question, many wordings

This pattern arrives in many forms, and every form asks for the same 3 things: a real Spark job you changed, what was wrong with it, and evidence that the change helped. Train your ear on the shape under the wording, because the interviewer will rarely say 'war story'.

All of these are the same question

When did you last speed up a slow job, and how?
What is the hardest performance problem you have debugged?
Walk me through a pipeline you are proud of.
A job of yours missed its SLA. What did you do?
Have you ever cut the cost of a Spark job?

The last 2 sound like different topics. They are the same request: an SLA miss is a runtime problem, and a Spark cost cut almost always comes from fewer bytes shuffled or fewer executor-hours. A well-chosen story can answer all 5.

What gets graded

The interviewer checks 3 things, in this order. First, is the story real: did you do the work, and can you describe what you saw while doing it? Second, is the fix correct: does it remove the cause you found? Third, can you explain why it worked in terms of how Spark ran the job? A story that clears all 3 with a modest result beats a dramatic one that stalls on the third, because the third is where the follow-up questions go.

Choosing the story

✓Stories that hold up

You made the change yourself and measured it
The cause is a Spark mechanism: a shuffle, a join strategy, skew or a partition count
Before and after come from the same job on comparable input
You can say what the Spark UI showed before you touched the code

✗Stories that crack under follow-ups

A teammate found the cause and you deployed the fix
The fix was a bigger cluster, and nobody knows why it helped
The only evidence is that it felt faster
A rewrite of 20 jobs where no single change can be named

With several candidates, keep the one whose cause you can trace in the plan, even if its speedup is smaller. A job that went from 40 minutes to 28 is a good story when you can account for the 12 minutes. Speed is also only one kind of result: a job that failed with out-of-memory errors on 6 nights out of 30, and then on none, is a war story with a number of its own.

Choose the headline figure first

The listener carries a single figure out of your answer, so choose it before the interview and put it in your opening sentence. Runtime is the usual pick because everyone understands minutes, and a deadline turns minutes into a business result. Bytes shuffled, cost per run or failure rate also work. Choose one for the headline and save the rest for the comparison later.

94 min

nightly runtime before the fix

21 min

nightly runtime after the fix

05:00

dashboard refresh deadline

Those figures come from the story this lesson follows all the way to the outro. Priya's job rebuilt regional revenue every night, and the dashboards that read it refreshed an hour after it started. Each later section adds detail to the same job.

Fact	Value
Job	store_revenue_nightly
Output	Revenue per region per day, last 90 days
Large input	orders: 9 billion order lines, 410 GB of Parquet
Small input	stores: 1,200 rows, 3 MB
Cluster	40 executors, 4 cores each
Window	Starts 04:00, dashboards refresh 05:00
Runtime	94 min before, 21 min after
Cause	Broadcast joins disabled by a shared config template

Priya's headline: 'Our nightly store revenue job took 94 minutes against a 60-minute window. I found a config line forcing a shuffle join against a 3 MB table, took it out, and the job now runs in 21 minutes.' It names the job, the before, the after and the cause in under 40 words, and it leaves the interviewer wanting the middle.

Compare it with the version most people give: 'I worked on optimizing some Spark jobs at my last company and made them a lot faster.' It has no job, no number and no cause, so every follow-up has to dig for what the first sentence should have handed over.

TIP

Say the before and the after in the same unit. '94 to 21 minutes' lands at once. '4.5 times faster' makes the listener do arithmetic and miss your next sentence.

Try it now

▸List 3 stories from your own work where a job got faster, cheaper or stopped failing.
▸Next to each, write its cause in Spark terms in a single line. Cross out any you cannot.
▸Take the survivor and write its headline: the job, the before, the after, the cause.
▸Say the headline aloud with a timer. Under 12 seconds is the target.

Get to Your Action Fast

Daily Life

Interviews

Where the first minutes go

Many war stories are lost before the engineering starts. The candidate covers the company, the team, the stack and the job's history, and the interviewer, who has budgeted a few minutes for the whole answer before follow-ups, hears none of the work being graded. Context earns its place only when the listener needs it to follow your action. For a Spark speedup that is 2 facts: what the job produces, and why its runtime mattered.

Segment	Slow opening	Fast opening
Headline sentence	none	8 s
Company and team	60 s	left out
Tech stack	50 s	named only where the fix touches it
History of the job	70 s	left out
Who uses the output	60 s	a clause: the 05:00 dashboards
Time before the action	240 s	28 s
Action starts at	not reached	0:28

The slow column is the opening Priya gave in the interview from the intro. At 4 minutes she still had not reached the change, and the interviewer had to ask for it.

The fast opening, word for word

Priya rebuilt her opening as the headline sentence from the last section followed by 2 sentences of context. Here is the context she says after the headline, and then she stops setting the scene.

Context in 2 sentences

'The job rebuilds revenue per region per day for the last 90 days, roughly 9 billion order lines. The regional dashboards read its output at 05:00 and it was finishing around 05:34, so managers opened yesterday's numbers every morning. Here is what I did.'

Leave out the company name, the team size, the scheduler and the year. If the interviewer wants any of those, they will ask, and answering a question in 5 seconds costs less than a paragraph they did not need. The 2 sentences keep the fact an interviewer will use later: the deadline, which is what makes 94 minutes a problem and 21 a fix.

Tell the action as a chain of evidence

The action is what the interviewer is scoring, so tell it in the order you lived it, and tie each step to a place in the Spark UI and the figure it showed. That proves you investigated, and every figure becomes a handle the interviewer can pull with a follow-up.

Noticed: the Jobs tab put the nightly job at 94 minutes, past the 05:00 refresh on every run that month.

Looked: in the Stages tab, one stage ran 68 minutes with 140 GB of shuffle read and 61 GB spilled to disk.

Found: the SQL tab showed a SortMergeJoin, fed by an Exchange from orders and another from the 3 MB stores.

Read: an Exchange node is a shuffle, so both inputs were being repartitioned across the network.

Asked why: 3 MB is far under the 10MB broadcast default, so I opened the Environment tab.

Cause: spark.sql.autoBroadcastJoinThreshold was -1, inherited from a shared config template.

Changed: dropped the inherited setting from this job and added a broadcast hint on stores.

Measured: 7 runs on each side, comparing runtime, plan and output.

Read the chain back and notice that the plan did the diagnosing. The Stages tab said where the time went, the SQL tab said what kind of join was running, and the clash between a tiny table and a shuffle on both join inputs pointed at a setting. That reasoning, spoken out loud, is the strongest minute of the answer.

	template: batch_default
	spark_conf:
	spark.executor.instances: 40
	spark.executor.cores: 4
	spark.executor.memory: 16g
	spark.sql.shuffle.partitions: 200
	spark.sql.autoBroadcastJoinThreshold: -1

Say 'I' for your part

Interviewers listen for 'we' in the action and follow it with 'what was your part?' Use 'I' for what you did: the investigation, the change, the comparison. Use 'we' for what the team decided, such as when to roll the change out. If a teammate found the cause, the story can still work: tell the part you owned, such as the measurement or the reconciliation, and name who found the cause.

TIP

Keep a second story ready. If the interviewer asks for a time something failed, or has heard a broadcast-join story twice that day, you can switch without inventing details on the spot.

Your turn

▸Write your context in 2 sentences and under 45 words, keeping the deadline.
▸Write your action as a chain: noticed, looked, found, changed, measured.
▸Beside each link, write the Spark UI tab and the figure it showed.
▸Time headline plus context aloud and cut until the action starts before 0:30.

The Before and After

Daily Life

Interviews

Two measurements of one job

A before-and-after compares the same job on comparable input, read from the same place each time. A before from a run over 60 days of data against an after over 90 says nothing, and an interviewer who asks 'were those the same input?' will catch it. Priya took 7 nightly runs on each side, every one rebuilding 90 days, and quoted the median.

Night	Before (min)	After (min)
1	91	20
2	97	22
3	94	21
4	88	19
5	103	24
6	92	21
7	96	20
Median	94	21

The spread matters as much as the median. Before runs ranged from 88 to 103 minutes and after runs from 19 to 24, so the slowest run after the fix still beat the fastest run before it by 64 minutes. When the 2 ranges do not overlap, you can say 'every run after the change beat every run before it', and nightly variation cannot explain the result. Quoting 103 against 19 would have claimed a bigger win than the job delivered, and an interviewer who asks about variance would find it.

Metric	Before	After	Where it is read
Runtime, median of 7 runs	94 min	21 min	Jobs tab, Duration
Join strategy	SortMergeJoin	BroadcastHashJoin	SQL tab, plan graph
Shuffle exchanges	3	1	SQL tab, Exchange nodes
Stages in the main job	4	2	Jobs tab, job detail
Slowest stage	68 min	19 min	Stages tab, Duration
Shuffle write	140 GB	about 1 MB	Stages tab, Shuffle Write
Spill (Disk)	61 GB	0	Stage detail, summary metrics

Runtime is the headline, and the other rows explain it. Interviewers trust a runtime claim when the rows under it move the way the mechanism predicts: fewer exchanges, less shuffle written, no spill.

The old plan, on a small sample

The animation runs the old job's shape on 12 order lines and 4 stores, each table read as 3 partitions. It groups by region alone to keep the board small; the real job grouped by region and day. With broadcasting off, Spark must bring each order line to the same task as its store, so both tables move. Watch both inputs get shuffled by store_id, then a third shuffle as groupBy brings each region to a single task.

1spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
2 
3revenue = (
4    orders.join(stores, "store_id")
5    .groupBy("region")
6    .agg(F.sum("amount").alias("revenue"))
7)
8revenue.collect()

orders12 rows

order_id

store_id

amount

101

102

101

103

104

102

103

101

104

102

103

104

stores4 rows

store_id

region

101

North

102

South

103

North

104

West

Preparing the animation

That run has 3 shuffle exchanges and 4 stages: a stage to scan each table, a stage to join and pre-sum, and a stage to finish the sums. At full size, the orders exchange is the one that carried the 140 GB in the table's Shuffle write row, and the 61 GB of spill came from sorting it. The last section walks through how.

The new plan, same sample

Here is the same sample with the broadcast hint. Spark collects stores on the driver and sends a copy to every executor, so each orders task looks up its stores locally and joins inside the scan. Watch the order rows stay on their tasks until the groupBy.

1revenue = (
2    orders.join(F.broadcast(stores), "store_id")
3    .groupBy("region")
4    .agg(F.sum("amount").alias("revenue"))
5)
6revenue.collect()

orders12 rows

order_id

store_id

amount

101

102

101

103

104

102

103

101

104

102

103

104

stores4 rows

store_id

region

101

North

102

South

103

North

104

West

Preparing the animation

One shuffle exchange and 2 stages remain. The shuffle left over carries partial sums, a row per region and day from each scan task, about 1 MB for the whole run. The broadcast has a price too: 3 MB sent to 40 executors is roughly 120 MB of network traffic per run, against 140 GB before.

Show the numbers did not change

A faster job that writes different numbers is a regression, and a correct answer says you checked. Swapping the join strategy should not change results, but 'should' is a claim and a query is evidence. For a week, Priya wrote the new job's output to a second table, store_revenue_candidate, and compared it with the old one, region by region and day by day.

	SELECT
	COALESCE(o.region, n.region) AS region,
	COALESCE(o.order_date, n.order_date) AS order_date,
	o.revenue AS old_revenue,
	n.revenue AS new_revenue
	FROM store_revenue AS o
	FULL OUTER JOIN store_revenue_candidate AS n
	ON o.region = n.region
	AND o.order_date = n.order_date
	WHERE o.revenue IS DISTINCT FROM n.revenue

The FULL OUTER JOIN keeps rows that exist in only one table, and IS DISTINCT FROM treats a NULL against a value as a difference, so a region missing from either version shows up as well as a changed sum.

Know what a failing result looks like before you need one. Had the candidate job lost one region on one night and summed another differently, the query would have returned rows like these:

region	order_date	old_revenue	new_revenue
West	2024-03-10	18240.50	NULL
North	2024-03-11	22915.00	22890.00

A NULL on one side means the row exists in only one version, which points at a filter or a join that dropped data. Two values that differ point at the aggregation or at rows counted twice. A difference of a few cents on many rows usually means a column type changed, such as DECIMAL becoming DOUBLE.

✓Do

Quote each metric with the Spark UI tab it came from
Compare several runs on the same input size
Name the shuffle that remains after the fix
Show the output matched with a reconciliation query

✗Don't

Pit one lucky run against one bad run
Give a ratio like 4.5x with no minutes behind it
Claim savings you never measured
Skip spill when the slow stage was spilling

Your turn

▸Write your before-and-after as a table: runtime, exchanges, shuffle write, spill.
▸Next to each figure, note the Spark UI tab you would read it from.
▸Decide how you would show the output did not change, and write that query.
▸If a number is missing, say in the interview that you did not measure it.

The System Tour

Daily Life

Interviews

When they ask you to walk through the system

A common first follow-up is 'can you describe the system around that job?' The interviewer is placing your fix: what feeds the job, how large each input is, where the output lands and who waits for it. A tour of about 30 seconds that names the sizes lets them see the problem before you explain it, which is the strongest place a war story can put a listener.

Sources

Raw

Processing

Serving

hourly order-line files

Store tills

Parquet

orders

410 GB for 90 days

stores

200 rows3 MB

Spark

store revenue nightly

40 executorsPARTITIONS200SLAdone by 05:00

540 rows per run

store revenue

refresh at 05:00

Regional dashboards

Order lines flow from the tills into orders every hour. Each night the job reads 90 days of orders plus all of stores and overwrites store_revenue, which the dashboards read at 05:00.

Six boxes cover it. The tour leaves out the file layout, the scheduler and the monitoring, because none of them touch the join. Add a box only when the fix depends on it.

Sizes are what make the tour useful

A tour without sizes is a list of names. The fact that unlocks Priya's story is the gap between the 2 inputs to the join: 410 GB on one side, 3 MB on the other. An engineer who hears those 2 sizes next to the words 'sort-merge join' can predict the fix before Priya names it.

orders

order_line_idPKBIGINT

order_dateDATE

store_idFKINTEGER

product_idBIGINT

quantityINTEGER

amountDECIMAL(12,2)

stores

store_idPKINTEGER

store_nameVARCHAR

regionVARCHAR

opened_onDATE

Say the join key and its direction as well: many order lines to a single store. That relationship is part of why the broadcast is safe. Each order line matches at most one store, so the join adds the region column without multiplying rows, and the joined data stays the size of orders.

When you do not remember a size

If you no longer have the sizes written down, recover them before the interview. The Stages tab shows bytes read in the Input column of each scan stage, and the scan node in the SQL tab reports its output rows and the size of files read. When you have only one of the 2, derive the other: about 100 million order lines a day for 90 days is 9 billion lines, and 410 GB spread over 9 billion lines is roughly 46 bytes per compressed line. A rounded figure such as 'over 400 GB' serves the tour well. A missing one leaves the interviewer unable to see why the join was the problem.

The tour, in the order data flows

Sourcestore tillsHourly files of order lines, about 100 million lines a day across 1,200 stores.
Inputsorders and storesorders holds 410 GB for the 90 days the job reads; stores is a 3 MB dimension, a row per store.
The jobstore_revenue_nightlyJoins orders to stores on store_id and sums amount by region and day, on 40 executors with 200 shuffle partitions.
Outputstore_revenueSix regions times 90 days, 540 rows, overwritten each night.
Consumerregional dashboardsRefresh at 05:00, an hour after the job starts, which sets the deadline.

Upstream to downstream is the order to use, because it is the order the interviewer will draw in their head. Starting at the dashboards and working backwards forces them to hold the whole picture before any of it connects.

Finish on the box that shuffled

End the tour by pointing at where the data moved: 'The join inside the job is where it shuffled, and that is the box I changed.' That sentence ties the system to your action and hands the interviewer the question you want next: why that box?

✓A tour that helps

Names each input with its size
Says the deadline once
Marks where the shuffle happens
Stops at the consumer

✗A tour that stalls

Lists tools with no sizes
Explains the company's data strategy
Covers every job in the pipeline
Never says where the fix sits

TIP

Draw the tour if there is a whiteboard or a shared document: 6 boxes, a size written under each, a mark on the join. Interviewers come back to a drawing during follow-ups, and every return is a chance to repeat your number.

Your turn

▸Draw your story's system in at most 6 boxes.
▸Write the size under each input: rows, bytes or both.
▸Mark the transformation you changed and the shuffle inside it.
▸Give the tour aloud in 30 seconds, upstream to downstream.

"Why Did That Work?"

Daily Life

Interviews

The follow-up that decides the answer

Most war stories end with some version of 'why did that work?' It separates a fix you understood from one you copied. The answer is a mechanism with 3 parts: where the data moved before, where it moves now, and what that movement cost.

Before: both tables travel

A sort-merge join needs each order line on the same task as its store. With broadcasting off, Spark got there by hash-partitioning both tables on store_id into 200 partitions. Each scan task wrote its rows to local disk, split by partition. Each of the 200 reducer tasks then fetched its partition from every scan task, sorted it by store_id and merged it with the matching stores rows. For orders that meant 140 GB written, 140 GB fetched and roughly 700 MB sorted per task, which spilled.

After: only the small table travels

A broadcast hash join brings the small table to the data. The driver collects stores, sends a copy to each of the 40 executors, and each executor builds an in-memory hash table keyed by store_id. Every orders scan task probes that hash table as its rows stream past. No order line is written or sent anywhere until the groupBy, and by then each task has collapsed its rows into a few partial sums.

✓Sort-merge join, before

Moves both tables: 140 GB of orders, plus stores
Adds 2 exchanges for the join
Sorts every order line by store_id
Risk: reducer tasks spill when partitions are large
Fits: 2 large tables

✗Broadcast hash join, after

Moves stores only: a copy per executor
Adds no exchange for the join
Sorts nothing; probes a hash table
Risk: the small side must fit in driver and executor memory
Fits: a small table that stays small

The questions that come next

Expect 2 or 3 of these after the mechanism, and prepare a 2-sentence answer to each for your own story.

Why not add executors?hardwareMore executors split the 140 GB over more tasks, but every byte is still written, fetched and sorted, and the bill grows with the cluster. Removing the movement beat spreading it.
Why not raise the shuffle partitions?spillAt 800 partitions each reducer sorts about 175 MB instead of 700 MB, so spill shrinks. The 140 GB still crosses the network, so it eases the symptom and keeps the cause.
What is the shuffle that is left?the groupByThe sum by region and day. Each scan task sends its partial sums, about 1 MB in total, and 200 tasks finish them.
What if stores keeps growing?headroomIt gains a few dozen rows a year. At 3 MB it is about a third of the default threshold and far below what driver or executor memory can hold.
Why was broadcasting off at all?historyAnother job hit an out-of-memory error during a broadcast, and the template author disabled automatic broadcasts for everyone. I raised it with the template's owners, since changing it for every job needed its own testing.
Did the results change?correctnessNo. A week of side-by-side output, reconciled by region and day, differed in 0 rows.

Predict the plan before you claim it

Interviewers sometimes change your code and ask what happens. Practice that by predicting the plan before running it. This version has no hint and no threshold setting, and it sorts the result by revenue. The stores sample is far below 10MB. Decide before you press play: how many shuffle exchanges, and how many stages?

1revenue = (
2    orders.join(stores, "store_id")
3    .groupBy("region")
4    .agg(F.sum("amount").alias("revenue"))
5    .orderBy(F.desc("revenue"))
6)
7revenue.collect()

orders12 rows

order_id

store_id

amount

101

102

101

103

104

102

103

101

104

102

103

104

stores4 rows

store_id

region

101

North

102

South

103

North

104

West

Preparing the animation

Two exchanges and 3 stages. The join broadcasts without a hint because stores is under the default threshold, so it adds no shuffle. The groupBy shuffles partial sums by region. The orderBy adds a range shuffle and a stage: Spark samples the revenue values to pick boundaries, then sends each row to the task that owns its range, so the collected rows arrive in order. Cutting the sorted result to its first few rows would let Spark run a top-N and skip that range shuffle.

TIP

If an interviewer asks why something worked and you do not know, say what you would check: 'I would compare the SQL tab plans before and after.' A plan you would look at passes. A guess stated as fact fails at the next probe.

Your turn

▸Write the mechanism of your fix in 3 sentences: what moved before, what moves now, what it cost.
▸Pick 4 follow-ups from the list above and answer each for your own story in 2 sentences.
▸Change a single line of your job's code on paper and predict its exchanges and stages.
▸Explain the fix aloud to someone who has not seen the job. Note where they get lost.

❯❯❯PUTTING IT ALL TOGETHER

> A food delivery marketplace runs a nightly courier pay job that joins 30 days of delivery events, about 1.2 TB, to a table of pay rates per city zone, then sums pay per courier per day for a payroll export at 06:00. When a rate-history column was added, the rates table grew to 14 MB, crossing the 10MB default threshold. Spark planned a sort-merge join, began shuffling the 1.2 TB of events every night, and runtime went from 50 minutes to 130, past the export. The engineer added a broadcast hint on the 14 MB table and runtime fell to 46 minutes. Months later she is asked in an interview about a time a job missed its SLA.

One story, one number: '130 minutes back to 46, inside the 06:00 payroll export' is her headline, and the cause is a join strategy, which holds up under follow-ups.

Get to the action: her context is the job's output and the deadline. Her action opens on the SQL tab showing a SortMergeJoin where last month's plan had a BroadcastHashJoin, then the Stages tab showing the event shuffle.

Before and after: 3 shuffle exchanges fall to 1, the 1.2 TB event shuffle disappears, and a reconciliation of pay per courier per day finds 0 differences, which matters more than usual when the output is people's wages.

System tour: delivery events at 1.2 TB, zone rates at 14 MB, the job, the payroll export. The 2 sizes side by side tell the listener what the fix will be.

Why it worked: a 14 MB table now goes to every executor once while the 1.2 TB table stays where it was read. The follow-up 'what if rates keep growing?' has an answer ready: the hint needs a size check before the table outgrows executor memory.

KEY TAKEAWAYS

Every wording of 'tell me about a slow job' asks for a real story with a cause you can trace in the Spark plan.

Open with a single sentence holding the job, the before, the after and the cause, with the headline number in it.

Keep context to 2 sentences, the job's output and its deadline, so your action starts before 30 seconds.

Tell the action as evidence: the Spark UI tab, the figure it showed, the conclusion, the change.

Back the runtime with the rows that explain it: exchanges, shuffle write, spill, and a reconciliation showing the output did not change.

Tour the system upstream to downstream with a size on every input, and finish on the box that shuffled.

Answer 'why did that work?' with where the data moved before, where it moves now, and what each movement cost.

Tell a Spark speedup story so the interviewer hears the number, the shuffle you removed and why it worked.

Category: Spark
Difficulty: beginner
Duration: 15 minutes
Challenges: 3 hands-on challenges

Topics covered: One Story, One Number, Get to Your Action Fast, The Before and After, The System Tour, "Why Did That Work?"

Lesson Sections

One Story, One Number (concepts: Spark UI Diagnosis)
One question, many wordings This pattern arrives in many forms, and every form asks for the same 3 things: a real Spark job you changed, what was wrong with it, and evidence that the change helped. Train your ear on the shape under the wording, because the interviewer will rarely say 'war story'. The last 2 sound like different topics. They are the same request: an SLA miss is a runtime problem, and a Spark cost cut almost always comes from fewer bytes shuffled or fewer executor-hours. A well-ch
Get to Your Action Fast (concepts: Broadcast Join)
Where the first minutes go Many war stories are lost before the engineering starts. The candidate covers the company, the team, the stack and the job's history, and the interviewer, who has budgeted a few minutes for the whole answer before follow-ups, hears none of the work being graded. Context earns its place only when the listener needs it to follow your action. For a Spark speedup that is 2 facts: what the job produces, and why its runtime mattered. The slow column is the opening Priya gave
The Before and After (concepts: Broadcast Join)
Two measurements of one job A before-and-after compares the same job on comparable input, read from the same place each time. A before from a run over 60 days of data against an after over 90 says nothing, and an interviewer who asks 'were those the same input?' will catch it. Priya took 7 nightly runs on each side, every one rebuilding 90 days, and quoted the median. The spread matters as much as the median. Before runs ranged from 88 to 103 minutes and after runs from 19 to 24, so the slowest
The System Tour (concepts: Pipeline Monitoring)
When they ask you to walk through the system A common first follow-up is 'can you describe the system around that job?' The interviewer is placing your fix: what feeds the job, how large each input is, where the output lands and who waits for it. A tour of about 30 seconds that names the sizes lets them see the problem before you explain it, which is the strongest place a war story can put a listener. Six boxes cover it. The tour leaves out the file layout, the scheduler and the monitoring, beca
"Why Did That Work?" (concepts: Broadcast Join)
The follow-up that decides the answer Most war stories end with some version of 'why did that work?' It separates a fix you understood from one you copied. The answer is a mechanism with 3 parts: where the data moved before, where it moves now, and what that movement cost. Before: both tables travel After: only the small table travels The questions that come next Expect 2 or 3 of these after the mechanism, and prepare a 2-sentence answer to each for your own story. Predict the plan before you cl