Dynamic Allocation
On a fixed cluster you choose spark.executor.instances at startup and pay for every executor until the application exits, busy or not. Dynamic allocation hands that count to the driver, which watches its own task queue, requests executors from the cluster manager when tasks wait, and releases executors that have nothing to run. Cluster size then follows the work in flight, stage by stage.
The number the driver targets
At any moment the driver knows how many tasks are running and how many are pending. One executor runs spark.executor.cores divided by spark.task.cpus tasks at once, usually 4 divided by 1. Divide running plus pending tasks by that and round up: the result is the executor count that would run every task in a single wave. Spark multiplies it by executorAllocationRatio (1 by default), caps it at maxExecutors and floors it at minExecutors. A ratio of 0.5 asks for half as many executors and accepts 2 waves instead of 1.
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("nightly_rollup")
.config("spark.executor.cores", "4")
.config("spark.dynamicAllocation.enabled", "true")
.config("spark.dynamicAllocation.shuffleTracking.enabled", "true")
.config("spark.dynamicAllocation.minExecutors", "2")
.config("spark.dynamicAllocation.maxExecutors", "50")
.config("spark.dynamicAllocation.executorIdleTimeout", "60s")
.config("spark.dynamicAllocation.schedulerBacklogTimeout", "1s")
.getOrCreate()
)Most of the risk sits in 3 of these lines. maxExecutors defaults to unlimited, so a stage of 10,000 tasks on 4-core executors asks for 2,500 of them. minExecutors defaults to 0, and every executor below the floor is one the next job waits for. The shuffle line is there because releasing an executor deletes the shuffle files it wrote, and later stages still read them. Spark runs dynamic allocation only when something protects those files. That is usually the external shuffle service serving them from each node, or shuffle tracking, which keeps an executor alive while a running job still needs its files. Decommissioning with shuffle migration also qualifies. Name one explicitly so the behaviour does not shift between Spark versions.
Ramp-up doubles every round
Once tasks have waited schedulerBacklogTimeout (1 second by default), the driver requests 1 more executor. If tasks are still waiting after sustainedSchedulerBacklogTimeout (also 1 second), it adds 2, then 4, then 8, doubling each round until it reaches the target. Starting at 1 keeps a short burst from setting off a large request, and doubling reaches a large target within a few rounds. Here is the rollup's 800-task first stage on the settings above, starting from the floor of 2:
| Round | Seconds after tasks queue | Executors added | Total requested |
|---|---|---|---|
| 1 | 1 | 1 | 3 |
| 2 | 2 | 2 | 5 |
| 3 | 3 | 4 | 9 |
| 4 | 4 | 8 | 17 |
| 5 | 5 | 16 | 33 |
| 6 | 6 | 17 | 50 |
800 tasks over 4 slots would need 200 executors, so the cap of 50 binds and round 6 adds 17 instead of 32. All 6 requests go out within 6 seconds. Idle nodes start executors within seconds, but a new machine takes a minute or more to arrive after its request. The machines from different rounds provision in parallel, so the ramp waits about one provisioning delay, and stage 1 runs on whatever has landed.
Drain-down waits on the idle timer
An executor that has run no task for executorIdleTimeout, 60 seconds by default, is released unless one of 3 things holds it: the minExecutors floor, shuffle files a running job still needs under shuffle tracking, or cached blocks, which have a timer of their own. Drain is slower than ramp by design: a full minute of quiet passes before anything leaves.
The animation runs a groupBy over 4 input partitions with 3 shuffle partitions. Count the tasks in each stage as it plays: 4 map tasks, then 3 reducers. That per-stage count, divided by slots per executor, is what the driver chases, so the target moves at every stage boundary that changes the task count.
from pyspark.sql import functions as F
totals = orders.groupBy("region").agg(F.sum("profit").alias("profit"))
totals.collect()| order_id | region | profit |
|---|---|---|
| 1 | East | 120 |
| 2 | West | 80 |
| 3 | East | 45.5 |
| 4 | North | 60 |
| 5 | West | 30 |
| 6 | North | 95 |
| 7 | East | 20 |
| 8 | West | 70 |
Scale it up to a job with a map stage of 800 tasks, a groupBy stage of 200, then a final stage of 8 tasks after repartition(8) that writes 8 files. With a cap of 50 the target runs 50, 50, then 2. The fate of the other 48 depends on how shuffle files are protected, because the final stage reads the 200-task stage's output from every executor that wrote it.
- External shuffle service
- The service on each node serves the 200-task stage's output
- The idle executors leave 60 seconds after their last task
- A 20-minute final stage runs on 2 executors
- Shuffle tracking
- The output lives only inside the executors that wrote it
- All 48 hold shuffle data the running job needs, so they stay
- A 20-minute final stage pays for 50 executors, roughly 15 extra executor-hours
A job runs with spark.executor.cores=4, minExecutors=0, maxExecutors=30 and the default backlog timeouts, on a cluster with idle capacity. A stage submits 200 tasks. How many executors has the driver requested 4 seconds after the tasks queue, and where does the target settle?
- Requested 7 at 4 seconds; the target settles at 50, since 200 tasks need 50 executors of 4 slots
- Requested 15 at 4 seconds; the target settles at 30, the maxExecutors cap
- Requested 15 at 4 seconds; the target settles at 50, since the cap limits only the first round
- Requested 30 after 1 second, because Spark asks for the full target at once when capacity is free
About this lesson
This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.