Speculative Execution
Concepts covered: Data Skew
A stage ends when its last task ends. If 199 of 200 tasks take 40 seconds and 1 lands on a machine with a failing disk, the stage lasts as long as that task, and every stage after it starts late. Speculative execution lets the TaskScheduler hedge: when a running task falls far behind its finished siblings, it launches a second attempt of the same partition on a different host, keeps whichever finishes first, and kills the other.
| Setting | Spark 3.5 default | What it controls |
|---|---|---|
| spark.speculation | false | Whether any copies are launched |
| spark.speculation.interval | 100ms | How often the TaskScheduler checks for slow tasks |
| spark.speculation.quantile | 0.75 | Share of a stage's tasks that must finish before checking starts |
| spark.speculation.multiplier | 1.5 | How many times the median finished duration a task must exceed |
| spark.speculation.minTaskRuntime | 100ms | Minimum runtime before a task can be copied |
Defaults move between releases, so check the configuration page for the version you run before relying on these numbers.
The rule applied to one stage
Stage 7 has 200 tasks, with speculation on and the defaults above. Nothing is checked until 150 tasks, 75% of 200, have finished. At that point the median duration of the finished tasks is 40 seconds, so the bar is 1.5 x 40 = 60 seconds, and every 100ms the TaskScheduler compares each running task's elapsed time with it.
| Task | Running for | Past 60 s? | Decision |
|---|---|---|---|
| 117 | 95 s | Yes | Copy queued for a core on another host |
| 44 | 61 s | Yes | Copy queued for a core on another host |
| 182 | 58 s | No | Checked again in 100ms |
| 9 | 41 s | No | Checked again in 100ms |
Task 117's original had read half its partition after 95 seconds, because its host's disk was serving reads slowly. Its copy, on a healthy executor, runs for 41 seconds and finishes at 136 seconds. The TaskScheduler marks partition 117 done and kills the original with the reason another attempt succeeded. Left alone, the original would have needed 190 seconds, so the copy saved 54.
Slow tasks a copy cannot fix
- A copy finishes sooner when
- The original's host has a failing disk or a saturated network link
- Another process on that node is starving the executor of CPU
- The slow task read about as much input as its siblings
- A copy wastes a core when
- The slow task's partition holds far more rows, so the copy repeats the same long read
- The cluster has no idle cores, so copies queue behind pending tasks
- The task writes to an outside system, which then receives every write twice
Check partition size first. A task 6 times slower than the median because its partition holds 6 times the rows gets a copy that reads those same rows, finishes no sooner, and holds a second core the whole time. On the stage page, compare the slow task's input size with the median input: a match makes the machine the suspect, and far more input means speculation cannot fix it.
Copies also run the same code twice at once. Spark's built-in file writers let only 1 attempt of each task commit its files, so tables Spark writes stay correct. Code inside a task that sends rows to an outside service, such as an HTTP call per row, runs in both attempts and sends everything twice.
Speculation is on with multiplier 1.5 and quantile 0.75. A stage has 80 tasks, 60 have finished, and their median duration is 20 s. Task 33 has run for 27 s and task 51 for 34 s. Which get a copy now?
- Task 51 only: the bar is 30 s and only task 51 has run past it.
- Both tasks: each has run longer than the 20 s median of the finished tasks.
- Neither task: checking starts only once more than 60 of the 80 tasks have finished.
- Task 51 only, and its copy runs on the same executor to reuse the rows read so far.
About this lesson
This section is part of the Execution Model: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.