Spot Instances
Spot capacity is a provider's spare machines, sold at a discount that providers advertise at up to 90 percent below on-demand prices, on condition that the provider can take them back. Notice runs from 30 seconds to 2 minutes depending on the provider, and then the node is gone with every executor on it and every file on its local disks.
What one reclaim takes
| Lost with the node | What Spark does | What it costs |
|---|---|---|
| Tasks running on it | Reschedules them on other executors | The work those tasks had done |
| Shuffle files it wrote | Reducers hit fetch failures; Spark reruns the lost map tasks, then retries the stage | Those map tasks plus a stage attempt |
| Its external shuffle service | Nothing to fall back on: the service died with the node | Same as the shuffle files |
| Cached blocks | Recomputes them from lineage at the next read | Lineage back to the last durable copy |
| The driver, if it ran there | The application ends | The whole run |
The shuffle row is where jobs die. Each fetch failure fails the reading stage's attempt, and spark.stage.maxConsecutiveAttempts (4 by default) aborts the job after 4 consecutive failed attempts of one stage. A reclaim wave that takes several nodes over a few minutes can fail the same stage repeatedly, because each retry reruns map tasks on nodes that may go next.
Each map task writes its shuffle output to the executor it ran on. In the join below broadcast is off, so both tables shuffle by title_id, and each of the 3 reducers fetches its share from every map task. Whichever node holds a map task's output is a dependency of every reducer.
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
joined = views.join(titles, "title_id")
joined.collect()| view_id | title_id | member_id |
|---|---|---|
| 1 | 10 | 501 |
| 2 | 20 | 502 |
| 3 | 10 | 503 |
| 4 | 30 | 501 |
| 5 | 40 | 504 |
| 6 | 20 | 505 |
| 7 | 30 | 502 |
| 8 | 10 | 506 |
| title_id | genre |
|---|---|
| 10 | Drama |
| 20 | Comedy |
| 30 | Documentary |
| 40 | Thriller |
If the executor that ran one views map task is reclaimed after the map stage, each reducer misses the rows that task wrote. Spark reruns that single map task, then the reducer attempts that failed. Lose 1 node out of 12 and about a twelfth of the map stage reruns; lose 4 at once and a third of it reruns while the reducers wait.
4 decisions for a spot fleet
- Driver on on-demand: Losing the driver ends the application. Every other node can come and go.
- An on-demand floor: Decide how long the job may take if every spot node vanishes, and keep enough on-demand executors to meet that.
- Several spot pools: Reclaims follow demand for one machine type in one zone, so 4 pools of 5 nodes lose 5 at a time where 1 pool of 20 can lose all 20.
- Smaller nodes: A node with 8 executors takes 8 executors' shuffle files with it; 2 nodes of 4 halve each loss.
Turn the notice into a migration
Graceful decommissioning, available since Spark 3.1, uses the notice. The executor stops accepting tasks and, with storage decommissioning on, copies its shuffle files and cached blocks to other executors, or to a fallback path in durable storage when no peer has room. Reducers then fetch from the new location and no map task reruns. This works only if the cluster manager passes the notice to Spark and the copy finishes inside the window, and an executor holding 30GB of shuffle output may not make it in 30 seconds. The config below also raises maxConsecutiveAttempts to 8, so a reclaim wave gets more attempts before the job aborts.
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("nightly_rollup")
.config("spark.decommission.enabled", "true")
.config("spark.storage.decommission.enabled", "true")
.config("spark.storage.decommission.shuffleBlocks.enabled", "true")
.config("spark.storage.decommission.rddBlocks.enabled", "true")
.config("spark.storage.decommission.fallbackStorage.path", "file:///mnt/shared/spark-fallback/")
.config("spark.stage.maxConsecutiveAttempts", "8")
.getOrCreate()
)For a job whose late stages sit on hours of earlier work, a reliable checkpoint() or a write and read-back at the midpoint puts a durable copy outside the cluster. A reclaim after that point recomputes from the copy instead of from the source. Spot lowers the price of each executor-hour and raises the expected rework; durable midpoints cap the rework.
Price the fleet, including the rework
Size the rollup's fleet at 25 executors, where it runs in 28 minutes, against a deadline of 45 minutes. Take on-demand executors at $0.25 an hour and spot at $0.08, and assume reclaims stretch any run with spot in it by 15 percent, to about 32 minutes.
| Fleet | Fleet price per hour | Expected runtime | Expected cost | If every spot node vanishes |
|---|---|---|---|---|
| 25 on-demand | $6.25 | 28 min | $2.92 | No effect |
| 15 on-demand + 10 spot | $4.55 | 32 min | $2.43 | 15 executors finish in 43 min |
| 8 on-demand + 17 spot | $3.36 | 32 min | $1.79 | 8 executors finish in about 69 min |
| 25 spot | $2.00 | 32 min | $1.07 | Stalls until spot capacity returns |
Every spot fleet is cheaper per run, but the last column tells you which ones you may use. If the 45-minute deadline must hold on the worst night, 15 on-demand executors is the smallest floor that keeps it, and spot can only add speed on top. If the deadline can slip on a bad night, the 8-executor floor saves another 64 cents a run.
A 3-hour job runs 40 spot executors on 5 large nodes, with its driver on on-demand and the external shuffle service enabled. In its last hour, reclaims hit 2 nodes a few minutes apart during the final shuffle stage, and the job aborts. Which change most directly keeps it alive?
- Spread the executors over more, smaller nodes in several spot pools, so each reclaim loses fewer shuffle files
- Keep the nodes and rely on the external shuffle service to serve their files after a reclaim
- Spread over smaller nodes, and set spark.stage.maxConsecutiveAttempts to 1 so failures surface sooner
- Move the driver onto spot as well, so it restarts lost stages closer to the executors
About this lesson
This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.