Scale-Down With Cached Data
One setting decides whether a cache can shrink a cluster: spark.dynamicAllocation.cachedExecutorIdleTimeout. Its default is infinity. An executor holding a single cached block never reaches the idle rule, however long it waits for a task. Spark keeps it because removal would destroy data you asked it to keep, and Spark has no way to know whether you will read that data again.
One cache freezes the cluster at its peak
Cached blocks live on the executors that ran the tasks computing them. Cache a 400-partition DataFrame while 50 executors are running, and roughly 8 partitions land on each. Every one of the 50 now holds a block, so every one is exempt from removal. Whatever size the cluster had when the cache filled is the size it keeps.
A weekly member-scoring job shows the pattern. It caches active titles, uses them in an early join, runs a 2-hour scoring step that never touches them, and reads them once more for a final report.
titles = spark.read.parquet("/warehouse/titles").filter("is_active").cache()
titles.count()
daily = events.join(titles, "title_id").groupBy("title_id", "event_day").count()
daily.write.mode("overwrite").parquet("/warehouse/daily_title_counts")
scores = score_members(members)
scores.write.mode("overwrite").parquet("/warehouse/member_scores")
report = scores.join(titles, "title_id").groupBy("genre").count()
report.write.mode("overwrite").parquet("/warehouse/genre_report")The count() fills the cache across all 50 executors. The scoring step's tasks occupy 4 executors for 2 hours, so the target drops to 4 and the other 46 run nothing. With the default timeout all 46 stay, because each holds part of titles. The table prices the 2-hour step at $0.25 per executor-hour under 3 choices.
| Choice | Executors held during scoring | Executor-hours | Cost of the step |
|---|---|---|---|
| Default, timeout infinite | 50 | 100 | $25.00 |
| `cachedExecutorIdleTimeout=5min` | 4, after 5 minutes | 11.8 | $2.96 |
| `unpersist()` after the daily write | 4, after 60 seconds | 8.8 | $2.19 |
- 92%
Both releasing choices pay a recompute: the final report reads titles again and rebuilds the missing partitions from lineage, which here means re-reading one filtered table. Releasing cached executors is cheap when the cached data is cheap to rebuild, and expensive when it sits at the end of hours of joins.
A notebook session uses dynamic allocation with every other setting at its default. At 9:00 it caches a 200-partition DataFrame while 40 executors are running, runs 3 queries on it by 9:20, and then sits open with no activity. How many executors does it hold at 15:00?
- Only minExecutors: every executor that ran no task after 9:20 passed the 60-second rule and left
- Holds 40 until 9:50, then drops to minExecutors once a default 30-minute cache expiry frees the blocks
- Holds 20, because Spark releases half of the cached executors and migrates their blocks to the rest
- All 40: each one holds cached blocks, and cachedExecutorIdleTimeout is infinite by default
4 ways to release them
- unpersist() after the last read that needs the blocks. The 60-second rule applies again at once. Best when the cache has a clear last use.
- A finite cachedExecutorIdleTimeout. Executors that run no task for that long leave with their blocks, and later reads recompute them. Set it longer than the gap between stages that read the cache.
- A write and read-back in place of the cache. The data lives in storage, so no executor holds it; a reliable checkpoint() behaves the same, while localCheckpoint() keeps blocks on executors and pins them like a cache.
- A finite timeout plus spark.decommission.enabled, spark.storage.decommission.enabled and spark.storage.decommission.rddBlocks.enabled: from Spark 3.1, an executor released this way copies its cached blocks to survivors first. Survivors need free storage memory, or the copies evict other blocks.
Locality sends each task that reads a cached partition to the executor holding it, so while stages keep reading the cache, those executors stay busy and neither idle timer fires on them.
With the external shuffle service, spark.shuffle.service.fetch.rdd.enabled=true serves DISK_ONLY cached blocks after their executor leaves, so executors whose blocks sit only on disk follow the 60-second rule. Blocks in memory still pin.
About this lesson
This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.