Scale-Down With Cached Data

One setting decides whether a cache can shrink a cluster: spark.dynamicAllocation.cachedExecutorIdleTimeout. Its default is infinity. An executor holding a single cached block never reaches the idle rule, however long it waits for a task. Spark keeps it because removal would destroy data you asked it to keep, and Spark has no way to know whether you will read that data again.

One cache freezes the cluster at its peak

Cached blocks live on the executors that ran the tasks computing them. Cache a 400-partition DataFrame while 50 executors are running, and roughly 8 partitions land on each. Every one of the 50 now holds a block, so every one is exempt from removal. Whatever size the cluster had when the cache filled is the size it keeps.

A weekly member-scoring job shows the pattern. It caches active titles, uses them in an early join, runs a 2-hour scoring step that never touches them, and reads them once more for a final report.

titles = spark.read.parquet("/warehouse/titles").filter("is_active").cache()
titles.count()

daily = events.join(titles, "title_id").groupBy("title_id", "event_day").count()
daily.write.mode("overwrite").parquet("/warehouse/daily_title_counts")

scores = score_members(members)
scores.write.mode("overwrite").parquet("/warehouse/member_scores")

report = scores.join(titles, "title_id").groupBy("genre").count()
report.write.mode("overwrite").parquet("/warehouse/genre_report")

The count() fills the cache across all 50 executors. The scoring step's tasks occupy 4 executors for 2 hours, so the target drops to 4 and the other 46 run nothing. With the default timeout all 46 stay, because each holds part of titles. The table prices the 2-hour step at $0.25 per executor-hour under 3 choices.

ChoiceExecutors held during scoringExecutor-hoursCost of the step
Default, timeout infinite50100$25.00
`cachedExecutorIdleTimeout=5min`4, after 5 minutes11.8$2.96
`unpersist()` after the daily write4, after 60 seconds8.8$2.19

Both releasing choices pay a recompute: the final report reads titles again and rebuilds the missing partitions from lineage, which here means re-reading one filtered table. Releasing cached executors is cheap when the cached data is cheap to rebuild, and expensive when it sits at the end of hours of joins.

A notebook session uses dynamic allocation with every other setting at its default. At 9:00 it caches a 200-partition DataFrame while 40 executors are running, runs 3 queries on it by 9:20, and then sits open with no activity. How many executors does it hold at 15:00?

4 ways to release them

Locality sends each task that reads a cached partition to the executor holding it, so while stages keep reading the cache, those executors stay busy and neither idle timer fires on them.

With the external shuffle service, spark.shuffle.service.fetch.rdd.enabled=true serves DISK_ONLY cached blocks after their executor leaves, so executors whose blocks sit only on disk follow the 60-second rule. Blocks in memory still pin.

About this lesson

This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.