Spot Instances

Spot capacity is a provider's spare machines, sold at a discount that providers advertise at up to 90 percent below on-demand prices, on condition that the provider can take them back. Notice runs from 30 seconds to 2 minutes depending on the provider, and then the node is gone with every executor on it and every file on its local disks.

What one reclaim takes

Lost with the nodeWhat Spark doesWhat it costs
Tasks running on itReschedules them on other executorsThe work those tasks had done
Shuffle files it wroteReducers hit fetch failures; Spark reruns the lost map tasks, then retries the stageThose map tasks plus a stage attempt
Its external shuffle serviceNothing to fall back on: the service died with the nodeSame as the shuffle files
Cached blocksRecomputes them from lineage at the next readLineage back to the last durable copy
The driver, if it ran thereThe application endsThe whole run

The shuffle row is where jobs die. Each fetch failure fails the reading stage's attempt, and spark.stage.maxConsecutiveAttempts (4 by default) aborts the job after 4 consecutive failed attempts of one stage. A reclaim wave that takes several nodes over a few minutes can fail the same stage repeatedly, because each retry reruns map tasks on nodes that may go next.

Each map task writes its shuffle output to the executor it ran on. In the join below broadcast is off, so both tables shuffle by title_id, and each of the 3 reducers fetches its share from every map task. Whichever node holds a map task's output is a dependency of every reducer.

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)

joined = views.join(titles, "title_id")
joined.collect()
views
view_idtitle_idmember_id
110501
220502
310503
430501
540504
620505
730502
810506
titles
title_idgenre
10Drama
20Comedy
30Documentary
40Thriller

If the executor that ran one views map task is reclaimed after the map stage, each reducer misses the rows that task wrote. Spark reruns that single map task, then the reducer attempts that failed. Lose 1 node out of 12 and about a twelfth of the map stage reruns; lose 4 at once and a third of it reruns while the reducers wait.

4 decisions for a spot fleet

Turn the notice into a migration

Graceful decommissioning, available since Spark 3.1, uses the notice. The executor stops accepting tasks and, with storage decommissioning on, copies its shuffle files and cached blocks to other executors, or to a fallback path in durable storage when no peer has room. Reducers then fetch from the new location and no map task reruns. This works only if the cluster manager passes the notice to Spark and the copy finishes inside the window, and an executor holding 30GB of shuffle output may not make it in 30 seconds. The config below also raises maxConsecutiveAttempts to 8, so a reclaim wave gets more attempts before the job aborts.

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("nightly_rollup")
    .config("spark.decommission.enabled", "true")
    .config("spark.storage.decommission.enabled", "true")
    .config("spark.storage.decommission.shuffleBlocks.enabled", "true")
    .config("spark.storage.decommission.rddBlocks.enabled", "true")
    .config("spark.storage.decommission.fallbackStorage.path", "file:///mnt/shared/spark-fallback/")
    .config("spark.stage.maxConsecutiveAttempts", "8")
    .getOrCreate()
)

For a job whose late stages sit on hours of earlier work, a reliable checkpoint() or a write and read-back at the midpoint puts a durable copy outside the cluster. A reclaim after that point recomputes from the copy instead of from the source. Spot lowers the price of each executor-hour and raises the expected rework; durable midpoints cap the rework.

Price the fleet, including the rework

Size the rollup's fleet at 25 executors, where it runs in 28 minutes, against a deadline of 45 minutes. Take on-demand executors at $0.25 an hour and spot at $0.08, and assume reclaims stretch any run with spot in it by 15 percent, to about 32 minutes.

FleetFleet price per hourExpected runtimeExpected costIf every spot node vanishes
25 on-demand$6.2528 min$2.92No effect
15 on-demand + 10 spot$4.5532 min$2.4315 executors finish in 43 min
8 on-demand + 17 spot$3.3632 min$1.798 executors finish in about 69 min
25 spot$2.0032 min$1.07Stalls until spot capacity returns

Every spot fleet is cheaper per run, but the last column tells you which ones you may use. If the 45-minute deadline must hold on the worst night, 15 on-demand executors is the smallest floor that keeps it, and spot can only add speed on top. If the deadline can slip on a bad night, the 8-executor floor saves another 64 cents a run.

A 3-hour job runs 40 spot executors on 5 large nodes, with its driver on on-demand and the external shuffle service enabled. In its last hour, reclaims hit 2 nodes a few minutes apart during the final shuffle stage, and the job aborts. Which change most directly keeps it alive?

About this lesson

This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.