A Question About Method
Concepts covered: Spark UI Diagnosis
The question arrives as a story. A nightly job that finished in 25 minutes last month now runs for nearly 3 hours, the code has not changed, and the pipeline owner wants it back under 30 minutes. The interviewer then stops talking. What they grade from that point is the order of what you say: whether you locate the slow part before you explain it, whether you explain it before you change it, and whether you say how you would know the change worked. A fix offered in the first sentence scores as a guess, however good the fix turns out to be, because the interviewer cannot tell whether you would have found a different cause.
Every disguise points at the same machinery. Spark splits each action into jobs, each job into stages at every shuffle, and each stage into 1 task per partition. A stage finishes when its last task finishes, and a job finishes when its last stage finishes, so the missing hours live somewhere specific: in one stage, and inside that stage in some set of tasks. A method is a route to that place. An answer without a route may land on the right fix and still fail the question, because the interviewer learns nothing about how you would handle the next slow job.
What a complete answer contains
The 5 moves, in the order they are spoken
- Locate: Open the Jobs tab, then the Stages tab sorted by duration. One stage usually holds most of the time.
- Measure: Read the task summary: median and max duration, shuffle read per task, spill, GC time.
- Diagnose: A lopsided max says a hot key. Even slow tasks with spill say memory pressure or too few partitions.
- Fix: Split the hot key out, raise the partition count, filter earlier, broadcast the small side.
- Verify: The stage's duration and its max task before and after, on the same input.
- A guess-first answer
- Opens with a fix: 'I would add more executors.'
- Names 1 cause and stops
- Cannot say which stage it would check
- Has no way to show the fix worked
- A method-first answer
- Opens with where the time is: 'I would find the slowest stage.'
- Names several causes and the symptom that separates them
- Picks the fix the symptom points to
- Ends with the measurement that proves it
| Opening sentence | What the interviewer hears |
|---|---|
| I'd add more memory. | A fix chosen before a diagnosis. |
| It's probably skew. | A likely cause, with no plan to confirm it. |
| I'd open the Spark UI and find the stage that holds most of the time. | A route to the cause, which they can follow with you. |
Why the order matters
Of the 5 moves, verify is the one candidates drop most often, and dropping it costs the most. A fix without a measurement is a change to production with no way to tell whether it helped, and a job that goes from 3 hours to 2 h 50 min after a memory increase looks like progress until someone compares the same stage on the same input. Locate comes first for a related reason: a change to a stage that takes 4 minutes of a 3-hour job can return 4 minutes at best, so the stage you pick caps what any fix can achieve. Say the moves in order and the interviewer can see each one resting on the one before it.
The rest of this lesson builds the method-first answer one move at a time: how to narrate the hunt for the stage when the interviewer gives you no screen, which causes to name and in what order, why the memory answer disappoints, and how to handle the commonest shape of all, where a single stage holds nearly all the time and a single task inside it holds nearly all of the stage.
About this lesson
This section is part of the Make It Faster lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.