BeginnerSpark · 14 min · 3 challenges
Make It Faster
Priya is 40 minutes into a data engineering interview at Netflix. The interviewer has finished with SQL and slides a different kind of question across the table: a nightly job that joins viewing events to a title catalog used to take 25 minutes and this week it runs for nearly 3 hours, nothing in the code changed, and the only question is what she would do. There is no laptop, no Spark UI, no logs. Priya's first instinct is to guess a cause, and the interviewer is hoping she won't. The question grades method. It wants to hear her open the UI in her head, find the stage that owns the time, read its task metrics, name the likely causes in order, pick a fix that matches the symptom, and say how she would prove it worked. This lesson builds that narration so the question sounds familiar wherever it appears.
What you will be able to do
Recognize the make-it-faster question in any disguise and answer it as a sequence of checks rather than a fix
Narrate the Spark UI from memory: Jobs tab, Stages tab, task summary, and the columns that separate causes
Name the usual causes of a slow stage, each with the symptom that confirms it, in the order they are cheapest to check
Explain what more executor memory changes and what it leaves untouched, with numbers
Diagnose a stage held by a single task, find the hot key, fix it, and state the measurement that proves the fix
The question arrives as a story. A nightly job that finished in 25 minutes last month now runs for nearly 3 hours, the code has not changed, and the pipeline owner wants it back under 30 minutes. The interviewer then stops talking. What they grade from that point is the order of what you say: whether you locate the slow part before you explain it, whether you explain it before you change it, and whether you say how you would know the change worked. A fix offered in the first sentence scores as a guess, however good the fix turns out to be, because the interviewer cannot tell whether you would have found a different cause.
The same question in 4 disguises
- ▸A job that used to take 20 minutes now takes 2 hours. Walk me through what you do.
- ▸This transformation is slow. How would you speed it up?
- ▸Here is a stage list from the Spark UI. What stands out to you?
- ▸A teammate says the fix is more executors. Do you agree?
Every disguise points at the same machinery. Spark splits each action into jobs, each job into stages at every shuffle, and each stage into 1 task per partition. A stage finishes when its last task finishes, and a job finishes when its last stage finishes, so the missing hours live somewhere specific: in one stage, and inside that stage in some set of tasks. A method is a route to that place. An answer without a route may land on the right fix and still fail the question, because the interviewer learns nothing about how you would handle the next slow job.
What a complete answer contains
The 5 moves, in the order they are spoken
- LocateWhich stage owns the wall clockOpen the Jobs tab, then the Stages tab sorted by duration. One stage usually holds most of the time.
- MeasureWhat the tasks inside it look likeRead the task summary: median and max duration, shuffle read per task, spill, GC time.
- DiagnoseWhich cause matches that shapeA lopsided max says a hot key. Even slow tasks with spill say memory pressure or too few partitions.
- FixOne change aimed at that causeSplit the hot key out, raise the partition count, filter earlier, broadcast the small side.
- VerifyThe same stage, re-run, comparedThe stage's duration and its max task before and after, on the same input.
✓A guess-first answer
- Opens with a fix: 'I would add more executors.'
- Names 1 cause and stops
- Cannot say which stage it would check
- Has no way to show the fix worked
✗A method-first answer
- Opens with where the time is: 'I would find the slowest stage.'
- Names several causes and the symptom that separates them
- Picks the fix the symptom points to
- Ends with the measurement that proves it
| Opening sentence | What the interviewer hears |
|---|---|
| I'd add more memory. | A fix chosen before a diagnosis. |
| It's probably skew. | A likely cause, with no plan to confirm it. |
| I'd open the Spark UI and find the stage that holds most of the time. | A route to the cause, which they can follow with you. |
Why the order matters
Of the 5 moves, verify is the one candidates drop most often, and dropping it costs the most. A fix without a measurement is a change to production with no way to tell whether it helped, and a job that goes from 3 hours to 2 h 50 min after a memory increase looks like progress until someone compares the same stage on the same input. Locate comes first for a related reason: a change to a stage that takes 4 minutes of a 3-hour job can return 4 minutes at best, so the stage you pick caps what any fix can achieve. Say the moves in order and the interviewer can see each one resting on the one before it.
The rest of this lesson builds the method-first answer one move at a time: how to narrate the hunt for the stage when the interviewer gives you no screen, which causes to name and in what order, why the memory answer disappoints, and how to handle the commonest shape of all, where a single stage holds nearly all the time and a single task inside it holds nearly all of the stage.
TIP
Say the first move out loud before anything else: 'The first thing I would do is find which stage owns the time.' That one sentence tells the interviewer you have a route, and it buys you the next 10 minutes.
The Spark UI has 1 screen per level of the hierarchy. The Jobs tab lists every action the program ran, with its duration and the stages it needed. The Stages tab lists every stage across all jobs, with its task count, its duration, its input size, and its shuffle read and shuffle write in bytes. Opening a stage shows the summary metrics for its tasks: min, 25th percentile, median, 75th percentile and max, for task duration, GC time, shuffle read size, and spill to memory and disk. Those 5 columns are the ones you read aloud, because the gap between median and max is the single number that separates the most common causes from each other.
A stage boundary is a shuffle. Spark runs everything it can without moving rows between machines inside one stage, then stops, writes each partition's rows to local disk grouped by destination, and starts the next stage by reading them back over the network. The program below has 1 shuffle, so it runs as 2 stages, and the animation shows which rows move and where they land.
orders.groupBy("region").sum("amount").collect()
Preparing the animation
Stage 1 reads the 3 input partitions and sums the amounts per region inside each one, so each partition writes at most 3 partial sums, grouped by the task that owns each region. The shuffle routes every partial sum for a region to the same task. Stage 2 adds the partial sums for each region and the driver collects 3 rows. That partial step is why an aggregate's shuffle write can be far smaller than its input. In the Stages tab the job appears as 2 lines, and the second line's shuffle read equals the first line's shuffle write, which is how you confirm that the two lines belong to the same boundary when the list is long.
The hunt, narrated
Here is the Stages tab for the job from the intro, as you would describe it from memory. The job scans a day of viewing events, scans the title catalog, joins them on title_id, aggregates hours watched per title, and writes the result. None of this is visible in the interview room, which is why the narration has to carry every number.
| Stage | What it does | Tasks | Duration | Shuffle write | Shuffle read | Spill (disk) |
|---|---|---|---|---|---|---|
| 0 | Scan viewing_events, prepare join key | 400 | 4 min | 38 GB | 0 | 0 |
| 1 | Scan title_catalog | 8 | 20 s | 1.2 GB | 0 | 0 |
| 2 | Join on title_id, aggregate | 200 | 2 h 51 min | 0.9 GB | 39.2 GB | 24 GB |
| 3 | Write output | 200 | 3 min | 0 | 0.9 GB | 0 |
Stage 2 owns the clock: 2 h 51 min of a 2 h 58 min job. The other 3 stages together take under 8 minutes, so a change to the scan, the catalog read, or the write cannot return more than 8 minutes however well it goes. Say that sentence in the interview. It is the moment the interviewer learns you can tell the 1 stage that matters from the 3 that do not. Then open stage 2 and read its task summary.
| Metric | Min | 25th | Median | 75th | Max |
|---|---|---|---|---|---|
| Duration | 22 s | 34 s | 41 s | 56 s | 2 h 48 min |
| Shuffle read | 60 MB | 110 MB | 135 MB | 160 MB | 12.4 GB |
| Spill (disk) | 0 | 0 | 0 | 0 | 24 GB |
| GC time | 0.4 s | 0.9 s | 1.2 s | 1.6 s | 4 min |
Read the shape. Half the tasks finish in 41 seconds and three quarters in under a minute, yet the stage takes 2 h 51 min because 1 task runs for 2 h 48 min. That task read 12.4 GB where its neighbours read 135 MB, about 90 times more, and it is the only one that spilled. One task holding the stage, with a shuffle read far above the median, is the signature of a hot join key, and the hunt has found both its stage and its task without a screen in the room.
//
If the stage list is long
A real job may have 30 stages across several actions. Sort the Stages tab by duration and add up the top 2 lines. If they cover 80% of the job, the hunt narrows to those 2. If the time is spread across 20 stages of a few minutes each, say so, because that shape points at a different cause, a slow scan or slow per-row code repeated in every stage, and a different fix than a single hot stage would.
The spoken version, in 6 sentences
- I would open the Jobs tab and find the job holding most of the time.
- In its stages I would sort by duration. Stage 2, the join, has 2 h 51 min of it.
- I would open stage 2 and read the task summary: median 41 s, max 2 h 48 min.
- One task read 12.4 GB of the 39 GB shuffle, and the median task read 135 MB.
- That is a hot value in title_id, so the next check is which value it is.
- After the fix I would re-run on the same input and expect the max task close to the median.
TIP
Name the tabs and the columns as you go: Jobs, Stages, task summary, median, max, shuffle read, spill. The vocabulary is the evidence that you have done this with a real screen.
When the interviewer says there is no UI, narrate what each screen would show anyway. The missing screen is the test, and the narration is the answer.
The moment after you find the stage, the interviewer asks what could make it slow. Two answers lose. Silence loses because it shows no map of the causes. A single cause loses because, if it is wrong, the conversation has nowhere to go. A short list of named causes, each paired with the symptom that would confirm it, wins because it hands the interviewer a menu: they pick one, you go deeper, and if the first is wrong the second is already on the table. The list is short. Seven entries cover nearly every slow stage in a batch job, and the order below is the order in which each is cheapest to check.
The usual suspects, cheapest check first
| Suspect | What the stage shows | First fix |
|---|---|---|
| Input grew | Scan stage input bytes up 3x on last week; all tasks evenly slower | Confirm the growth with the data owner; add a date filter or scale out rather than tune |
| Hot key | One task's duration and shuffle read far above the median; the rest finish fast | Find the hot value; split it out, filter it, or salt it |
| Too few shuffle partitions | Every task reads over 500 MB; spill on most of them | Raise `spark.sql.shuffle.partitions` so each task reads 100 to 200 MB |
| Too many tiny tasks | Thousands of tasks of 1 to 2 seconds; scheduling overhead dominates | Coalesce, or read fewer and larger files |
| Shuffle volume | Shuffle write in tens of GB for a result of a few MB | Filter and project before the join; broadcast a small side |
| Memory pressure | Spill on most tasks with even durations; GC time over 10% of task time | More partitions first, then fewer cores per executor, then more memory |
| Slow per-row code | Even slow tasks, little shuffle, no spill; a user function in the plan | Replace the function with built-in column operations |
Input growth comes first because it costs 1 comparison: this week's bytes out of the scan stage against last week's. A job whose input tripled and whose time tripled has no bug, and tuning it wastes an afternoon that a bigger cluster or a daily partition filter would save. The hot key comes second because 1 look at the task summary settles it: the max task next to the median. The remaining 5 each need a second number before you can tell them apart, so they come after.
Apply the list to stage 2 from the narration. Input growth: stage 0's shuffle write, the events it hands to the join, is 38 GB this week against 35 GB last week, a rise of under 10%, while the time grew sevenfold, which rules growth out as the main cause. Hot key: the max task read 12.4 GB against a median of 135 MB, which is the signature. Two checks, 2 numbers, and the list has done its work, with the other 5 suspects named and set aside in the interviewer's hearing.
200
shuffle partitions by default
10 MB
broadcast join threshold by default
128 MB
max bytes per scan partition
Those 3 defaults sit behind 3 of the suspects. A 40 GB shuffle over the default partition count gives each task about 200 MB, which fits a task's memory share. A 400 GB shuffle over the same count gives each task 2 GB, which spills. A catalog of 1.2 GB sits far above the broadcast threshold, so it is shuffled on the key rather than copied to every executor, and the join runs as a sort-merge join with both sides moved. A scan of 38 GB splits into a few hundred tasks as long as the files are splittable, which is why the scan stage rarely owns the clock.
Why saying them fast beats one good guess
The interviewer has a cause in mind, planted in the story. In the intro job it is a hot key that appeared in the data. If you name the hot key alone and are right, you pass the moment. If the planted cause was memory pressure, you have nothing left. If you name 4 causes with their symptoms in 30 seconds, you pass in both worlds, because the planted cause is on your list and you have told the interviewer how you would tell it apart from the others. Breadth here signals that you have debugged more than 1 slow job, and the symptom pairing signals that you would not try fixes at random on a production cluster.
✓Do
- Name 3 to 5 causes, each with the UI symptom that confirms it
- Check input growth before any tuning
- Order the list by how cheap each check is
- Say which number on the stage page separates the top 2 causes
✗Don't
- Stop at 1 cause without a way to confirm it
- List causes without the symptom that distinguishes them
- Name causes the stage metrics have already ruled out
- Spend the first minute on cluster size
Say the symptom, not only the name
Each suspect has a 1-line symptom you can say with it. Hot key: max task far above the median. Too few partitions: every task reads over 500 MB. Memory pressure: spill on most tasks with even durations. Volume: shuffle write far above the output size. Tiny tasks: thousands of tasks under 2 seconds. A cause named with its symptom is a hypothesis the interviewer can test with you. A cause named alone is a word.
Add more memory is the most common first answer to a slow job, and the interviewer hears it as a reflex. Memory in an executor does 1 thing for a running task: it holds the task's working set, the hash table or sorted buffer that a join or an aggregate builds, so the task avoids writing that buffer to disk. Spark reserves 300 MB of the heap, gives a fraction of the rest (0.6 by default) to execution and storage together, and divides that between the tasks running on the executor. When a task's buffer outgrows its share, it spills: it writes the buffer to local disk, continues, and merges the pieces later. More memory means fewer spills. It changes nothing else about the task: the rows it must process, the bytes it must read over the network, the code it must run on each row.
What memory fixes, and what it leaves alone
| Symptom | Does more memory help? | Why |
|---|---|---|
| 1 task far slower than the median | No | The hot task still reads its 12.4 GB alone; a bigger heap shortens its spill, and its row count stays the same |
| Most tasks spill, durations even | Yes | Each task's buffer now fits, so the spill disappears and the disk write with it |
| Shuffle write far above output size | No | The bytes still cross the network; memory does not reduce them |
| Thousands of 1-second tasks | No | Scheduling overhead has nothing to do with heap size |
| Input tripled since last week | No | Three times the rows need 3 times the work; memory only moves where the work happens |
| GC time over 10% of task time | Sometimes | A larger heap can lengthen each pause; more partitions or fewer cores per executor often do more |
Run the memory answer against stage 2. The job's executors have 8 GB each with 4 cores, so each task gets roughly 1 GB of execution memory. The median task reads 135 MB and never spills, so 199 of the 200 tasks have nothing to gain. The hot task reads 12.4 GB and spilled 24 GB over its 2 h 48 min. Doubling the executor to 16 GB gives that task roughly 2 GB, still a sixth of what it reads, so it still spills, somewhat less, and still runs for well over 2 hours. The stage that owned the clock still owns it, and the cluster now runs fewer executors per node for the same bill.
When memory is the right answer
- ▸Task durations are even: the max within 2x of the median
- ▸Spill shows on most tasks, rather than on 1
- ▸Shuffle read per task is modest, a few hundred MB at most
- ▸GC time is a visible share of task time on every task
- ▸No single value of the key dominates the input
That profile is memory pressure, and it is a real cause: a stage where every task reads 600 MB on a 1 GB share will spill on all 200 tasks, and the stage runs perhaps twice as long as it should. Even then, memory is the second fix to name. Raising the shuffle partition count from 200 to 800 drops each task to 150 MB on the same hardware, with no new bill, and the only cost is 600 more small tasks to schedule. The interviewer wants to hear that you know the free fix before the paid one.
The honest version of the memory answer
There is a case where memory is the first fix, and it is worth separating from the slow case: the job that dies. An executor lost with a container-killed message, or an out-of-memory error in the driver after a collect, is a size problem rather than a speed problem, and the stage metrics will show tasks that never finished rather than tasks that ran long. In the interview, ask which one the job did. A job that finishes late has a profile to read. A job that fails has a limit to raise, and even then the question of why one task needed that much memory, which is usually the same hot key, is worth a sentence before the limit goes up.
✓Saying 'add memory' first
- Signals a reflex rather than a reading of the stage
- Costs money on every run
- Leaves a hot key, shuffle volume and tiny tasks untouched
- Gives the interviewer nothing to probe
✗Saying it after the stage profile
- Signals that the symptom pointed there
- Comes after the free partition fix
- Names the task-level number it will change: spill
- Ends with the re-run that shows spill at 0
The cost side deserves 1 sentence in the room. A node with 64 GB fits about 7 executors at 8 GB each, once the operating system and per-executor overhead take their share, and about 3 at 16 GB. Doubling memory on a fixed cluster therefore cuts the executor count by more than half, and a stage with 200 evenly sized tasks then runs on fewer cores and gets slower. The memory answer can lose time as well as money, which is the part the reflex never mentions.
TIP
If you do name memory, pair it with the number it will change and the number it will not: 'Memory would reduce the spill on the hot task, but that task would still read 12.4 GB alone, so the stage would still take over 2 hours.'
The commonest shape in this question is the one from the narration: 1 stage holds nearly all the job, and inside it 1 task holds nearly all the stage. The 199 ordinary tasks finish within the first few minutes and their executors sit idle for nearly 3 hours waiting for 1 task, because the stage cannot end until every task ends and the next stage cannot start until the stage ends. The cluster is 99% idle and the bill is 100% running. The shape has 1 cause in a join or an aggregate: a key value that too many rows share, so the task that owns that value's partition receives all of them.
The animation below turns off broadcasting, so the join runs as a sort-merge join and both sides shuffle on the key, and it gives the events table a key value that most rows share. Watch which task receives those rows and when the stage finishes.
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)events.join(titles, "title_id", "left").count()
Preparing the animation
Seven of the 12 event rows carry no title_id. A left join keeps them, every one hashes to the same partition, and 1 task receives at least 7 of the 12 rows. Scale that ratio to a day of viewing events and the task that owns the empty key reads 12.4 GB while the median reads 135 MB. Without adaptive skew handling, the stage's duration is that task's duration however many executors the cluster has, because no other task can take a share of its partition. Adaptive execution, when enabled, can split a skewed join partition into several tasks, so 1 task of 2 h 48 min in the UI means it was off or did not apply. Splitting the empty rows out still matters: it removes their join work rather than spreading it.
Find the hot value
Before any fix, confirm which value is hot, because the fix depends on it. One aggregate over the join column answers the question, and it is the first query to run after the UI has pointed at the stage. Say the query in the interview. It is a checkable step rather than a hunch, and the interviewer can picture its result.
| title_id | event_rows | share of day |
|---|---|---|
| NULL | 61,400,000 | 31% |
| 80100172 | 2,900,000 | 1.5% |
| 81040344 | 2,100,000 | 1.1% |
| 80192098 | 1,800,000 | 0.9% |
| 70300500 | 1,500,000 | 0.8% |
The hot value is empty. A client release 3 days ago changed the app-open event: it used to carry the title the viewer last watched and now arrives with no title, so rows that always existed now share 1 key, and they make up 31% of the day. The input barely grew and the code did not change; what changed is how the rows spread across keys, which a byte count cannot see and the task summary can. These rows need no catalog lookup at all: a left join on an empty key matches nothing and returns the catalog columns as empty. So the fix removes them from the join, adds the empty columns directly, and unions the 2 halves back together.
A third of the stage's join work was for rows that needed none of it. After the change the join stage reads 26.8 GB instead of 39.2 GB, every task reads near the median, and the stage finishes when its slowest ordinary task does. The same input, re-run, gives the before-and-after that the method ends with, and the table is what you would describe in the room.
| Stage 2 | Before | After |
|---|---|---|
| Duration | 2 h 51 min | 6 min |
| Max task duration | 2 h 48 min | 58 s |
| Median task duration | 41 s | 39 s |
| Max task shuffle read | 12.4 GB | 190 MB |
| Spill (disk) | 24 GB | 0 |
| Whole job | 2 h 58 min | 13 min |
Read the top key before choosing the fix
- ▸
NULL, an empty string, 0 or -1 at the top is a placeholder for a missing value: filter it out of the join and pad its columns. - ▸A test account or a bot at the top is junk: drop its rows before the join, or upstream at the source.
- ▸A real title at the top is demand: its rows need the catalog, so salt that key or broadcast a small catalog.
- ▸Top keys at similar shares mean no hot key: return to the task summary and the suspect list.
When the hot value is a real key rather than an empty one, a live event title that 30% of viewers watched, filtering is wrong because those rows need the catalog. Broadcasting the catalog avoids the shuffle, but 1.2 GB is over 100 times the 10 MB threshold: the driver would collect it, and every 8 GB executor would hold a full copy plus its hash table beside 4 task shares of roughly 1 GB, which risks killed executors. Salting fits better: give the hot key's events a random suffix from 0 to 9, copy that 1 catalog row 10 times with each suffix, set suffix 0 on every other row, and join on key and suffix, so 10 tasks share the hot rows. Broadcast only if projecting the catalog to the columns the job needs shrinks it enough. Then say the measurement: the max task within 2x of the median on the re-run.
TIP
Close with the verification sentence every time: 'I would re-run on the same day's input and expect stage 2's max task within 2x of its median.' That sentence turns a fix into a claim the interviewer can test.
❯❯❯PUTTING IT ALL TOGETHER
> A streaming company's nightly job computes hours watched per title by joining a day of viewing events to the title catalog. After a client release starts logging app-open sessions with no title, 31% of the day's rows carry an empty key, the join stage balloons from minutes to nearly 3 hours, and the on-call engineer is asked in the morning stand-up what to do.
The method from section 0 keeps the engineer from reaching for a fix before the Stages tab says which of the 4 stages owns the time.
The narration from section 1 reads stage 2's task summary: median 41 s, max 2 h 48 min, 1 task reading 12.4 GB, the signature of a hot key.
The suspect list from section 2 rules out input growth with 1 comparison of scan bytes and confirms the hot key with 1 look at the max task.
Section 3 explains why doubling executor memory would have left that 1 task reading 12.4 GB alone and the stage still over 2 hours.
Section 4 finds the empty key with a count per title_id, routes those rows around the join, and verifies the stage at 6 minutes with the max task at 58 s.
KEY TAKEAWAYS
The question grades the order of your checks: locate the stage, measure its tasks, diagnose from the shape, fix the cause, verify on the same input.
Narrate the Spark UI by name even without a screen: Jobs tab, Stages tab sorted by duration, task summary with median and max.
Name several causes with the symptom that confirms each, cheapest check first, starting with whether the input grew.
More memory only reduces spill; it leaves a hot key, shuffle volume, tiny tasks and input growth untouched, and it costs executors per node.
A stage held by 1 task is a hot key: confirm the value with a count per key, keep it out of the shuffle, and prove it with the max task back near the median.
Find the stage, read its tasks, name the cause, pick the fix, prove it: the method a slow-job question grades.
- Category
- Spark
- Difficulty
- beginner
- Duration
- 14 minutes
- Challenges
- 3 hands-on challenges
Topics covered: A Question About Method, Narrate the Hunt for the Slow Stage, Name the Usual Suspects, "Just Add More Memory", One Stage Dominates
Lesson Sections
- A Question About Method (concepts: Spark UI Diagnosis)
The question arrives as a story. A nightly job that finished in 25 minutes last month now runs for nearly 3 hours, the code has not changed, and the pipeline owner wants it back under 30 minutes. The interviewer then stops talking. What they grade from that point is the order of what you say: whether you locate the slow part before you explain it, whether you explain it before you change it, and whether you say how you would know the change worked. A fix offered in the first sentence scores as a
- Narrate the Hunt for the Slow Stage (concepts: Spark UI Diagnosis)
The Spark UI has 1 screen per level of the hierarchy. The Jobs tab lists every action the program ran, with its duration and the stages it needed. The Stages tab lists every stage across all jobs, with its task count, its duration, its input size, and its shuffle read and shuffle write in bytes. Opening a stage shows the summary metrics for its tasks: min, 25th percentile, median, 75th percentile and max, for task duration, GC time, shuffle read size, and spill to memory and disk. Those 5 column
- Name the Usual Suspects (concepts: Spark UI Diagnosis)
The moment after you find the stage, the interviewer asks what could make it slow. Two answers lose. Silence loses because it shows no map of the causes. A single cause loses because, if it is wrong, the conversation has nowhere to go. A short list of named causes, each paired with the symptom that would confirm it, wins because it hands the interviewer a menu: they pick one, you go deeper, and if the first is wrong the second is already on the table. The list is short. Seven entries cover nearl
- "Just Add More Memory" (concepts: Spark UI Diagnosis)
Add more memory is the most common first answer to a slow job, and the interviewer hears it as a reflex. Memory in an executor does 1 thing for a running task: it holds the task's working set, the hash table or sorted buffer that a join or an aggregate builds, so the task avoids writing that buffer to disk. Spark reserves 300 MB of the heap, gives a fraction of the rest (0.6 by default) to execution and storage together, and divides that between the tasks running on the executor. When a task's b
- One Stage Dominates (concepts: Spark UI Diagnosis)
The commonest shape in this question is the one from the narration: 1 stage holds nearly all the job, and inside it 1 task holds nearly all the stage. The 199 ordinary tasks finish within the first few minutes and their executors sit idle for nearly 3 hours waiting for 1 task, because the stage cannot end until every task ends and the next stage cannot start until the stage ends. The cluster is 99% idle and the bill is 100% running. The shape has 1 cause in a join or an aggregate: a key value th