One Story, One Number

Concepts covered: Spark UI Diagnosis

One question, many wordings

This pattern arrives in many forms, and every form asks for the same 3 things: a real Spark job you changed, what was wrong with it, and evidence that the change helped. Train your ear on the shape under the wording, because the interviewer will rarely say 'war story'.

All of these are the same question

• When did you last speed up a slow job, and how?
• What is the hardest performance problem you have debugged?
• Walk me through a pipeline you are proud of.
• A job of yours missed its SLA. What did you do?
• Have you ever cut the cost of a Spark job?

The last 2 sound like different topics. They are the same request: an SLA miss is a runtime problem, and a Spark cost cut almost always comes from fewer bytes shuffled or fewer executor-hours. A well-chosen story can answer all 5.

What gets graded

The interviewer checks 3 things, in this order. First, is the story real: did you do the work, and can you describe what you saw while doing it? Second, is the fix correct: does it remove the cause you found? Third, can you explain why it worked in terms of how Spark ran the job? A story that clears all 3 with a modest result beats a dramatic one that stalls on the third, because the third is where the follow-up questions go.

Choosing the story

Stories that hold up

• You made the change yourself and measured it
• The cause is a Spark mechanism: a shuffle, a join strategy, skew or a partition count
• Before and after come from the same job on comparable input
• You can say what the Spark UI showed before you touched the code

Stories that crack under follow-ups

• A teammate found the cause and you deployed the fix
• The fix was a bigger cluster, and nobody knows why it helped
• The only evidence is that it felt faster
• A rewrite of 20 jobs where no single change can be named

With several candidates, keep the one whose cause you can trace in the plan, even if its speedup is smaller. A job that went from 40 minutes to 28 is a good story when you can account for the 12 minutes. Speed is also only one kind of result: a job that failed with out-of-memory errors on 6 nights out of 30, and then on none, is a war story with a number of its own.

Choose the headline figure first

The listener carries a single figure out of your answer, so choose it before the interview and put it in your opening sentence. Runtime is the usual pick because everyone understands minutes, and a deadline turns minutes into a business result. Bytes shuffled, cost per run or failure rate also work. Choose one for the headline and save the rest for the comparison later.

• 94 min
• 21 min
• 05:00

Those figures come from the story this lesson follows all the way to the outro. Priya's job rebuilt regional revenue every night, and the dashboards that read it refreshed an hour after it started. Each later section adds detail to the same job.

Priya's headline: 'Our nightly store revenue job took 94 minutes against a 60-minute window. I found a config line forcing a shuffle join against a 3 MB table, took it out, and the job now runs in 21 minutes.' It names the job, the before, the after and the cause in under 40 words, and it leaves the interviewer wanting the middle.

Compare it with the version most people give: 'I worked on optimizing some Spark jobs at my last company and made them a lot faster.' It has no job, no number and no cause, so every follow-up has to dig for what the first sentence should have handed over.

About this lesson

This section is part of the The War Story lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.