AdvancedSpark · 14 min · 13 challenges

Cluster Sizing: Advanced

Take the 2 a.m. job at Netflix that rebuilds every member's viewing history: it must land within 45 minutes, before morning dashboards read it. It runs on cheap spot nodes and grows from 2 executors to 50 in seconds. At 2:20 the cloud reclaims a node holding shuffle files its last stage needs. Whether the job finishes, and what each run costs, was decided in its config before it started.

Dynamic Allocation

Daily Life
Interviews

On a fixed cluster you choose spark.executor.instances at startup and pay for every executor until the application exits, busy or not. Dynamic allocation hands that count to the driver, which watches its own task queue, requests executors from the cluster manager when tasks wait, and releases executors that have nothing to run. Cluster size then follows the work in flight, stage by stage.

The number the driver targets

At any moment the driver knows how many tasks are running and how many are pending. One executor runs spark.executor.cores divided by spark.task.cpus tasks at once, usually 4 divided by 1. Divide running plus pending tasks by that and round up: the result is the executor count that would run every task in a single wave. Spark multiplies it by executorAllocationRatio (1 by default), caps it at maxExecutors and floors it at minExecutors. A ratio of 0.5 asks for half as many executors and accepts 2 waves instead of 1.

from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("nightly_rollup")
.config("spark.executor.cores", "4")
.config("spark.dynamicAllocation.enabled", "true")
.config("spark.dynamicAllocation.shuffleTracking.enabled", "true")
.config("spark.dynamicAllocation.minExecutors", "2")
.config("spark.dynamicAllocation.maxExecutors", "50")
.config("spark.dynamicAllocation.executorIdleTimeout", "60s")
.config("spark.dynamicAllocation.schedulerBacklogTimeout", "1s")
.getOrCreate()
)

Most of the risk sits in 3 of these lines. maxExecutors defaults to unlimited, so a stage of 10,000 tasks on 4-core executors asks for 2,500 of them. minExecutors defaults to 0, and every executor below the floor is one the next job waits for. The shuffle line is there because releasing an executor deletes the shuffle files it wrote, and later stages still read them. Spark runs dynamic allocation only when something protects those files. That is usually the external shuffle service serving them from each node, or shuffle tracking, which keeps an executor alive while a running job still needs its files. Decommissioning with shuffle migration also qualifies. Name one explicitly so the behaviour does not shift between Spark versions.

Ramp-up doubles every round

Once tasks have waited schedulerBacklogTimeout (1 second by default), the driver requests 1 more executor. If tasks are still waiting after sustainedSchedulerBacklogTimeout (also 1 second), it adds 2, then 4, then 8, doubling each round until it reaches the target. Starting at 1 keeps a short burst from setting off a large request, and doubling reaches a large target within a few rounds. Here is the rollup's 800-task first stage on the settings above, starting from the floor of 2:

RoundSeconds after tasks queueExecutors addedTotal requested
1113
2225
3349
44817
551633
661750
800 tasks over 4 slots would need 200 executors, so the cap of 50 binds and round 6 adds 17 instead of 32. All 6 requests go out within 6 seconds. Idle nodes start executors within seconds, but a new machine takes a minute or more to arrive after its request. The machines from different rounds provision in parallel, so the ramp waits about one provisioning delay, and stage 1 runs on whatever has landed.

Drain-down waits on the idle timer

An executor that has run no task for executorIdleTimeout, 60 seconds by default, is released unless one of 3 things holds it: the minExecutors floor, shuffle files a running job still needs under shuffle tracking, or cached blocks, which have a timer of their own. Drain is slower than ramp by design: a full minute of quiet passes before anything leaves.

The animation runs a groupBy over 4 input partitions with 3 shuffle partitions. Count the tasks in each stage as it plays: 4 map tasks, then 3 reducers. That per-stage count, divided by slots per executor, is what the driver chases, so the target moves at every stage boundary that changes the task count.

from pyspark.sql import functions as F
totals = orders.groupBy("region").agg(F.sum("profit").alias("profit"))
totals.collect()
orders8 rows
order_id
region
profit
1
East
120
2
West
80
3
East
45.5
4
North
60
5
West
30
6
North
95
7
East
20
8
West
70
Preparing the animation

Scale it up to a job with a map stage of 800 tasks, a groupBy stage of 200, then a final stage of 8 tasks after repartition(8) that writes 8 files. With a cap of 50 the target runs 50, 50, then 2. The fate of the other 48 depends on how shuffle files are protected, because the final stage reads the 200-task stage's output from every executor that wrote it.

✓External shuffle service
  • The service on each node serves the 200-task stage's output
  • The idle executors leave 60 seconds after their last task
  • A 20-minute final stage runs on 2 executors
✗Shuffle tracking
  • The output lives only inside the executors that wrote it
  • All 48 hold shuffle data the running job needs, so they stay
  • A 20-minute final stage pays for 50 executors, roughly 15 extra executor-hours
TIP
The Jobs page of the Spark UI has an event timeline marking each executor added and removed. Read it against the stage bars to see whether ramp-up lagged the first stage or executors outlived the work.
Multiple Choice

A job runs with spark.executor.cores=4, minExecutors=0, maxExecutors=30 and the default backlog timeouts, on a cluster with idle capacity. A stage submits 200 tasks. How many executors has the driver requested 4 seconds after the tasks queue, and where does the target settle?

Scale-Down With Cached Data

Daily Life
Interviews

One setting decides whether a cache can shrink a cluster: spark.dynamicAllocation.cachedExecutorIdleTimeout. Its default is infinity. An executor holding a single cached block never reaches the idle rule, however long it waits for a task. Spark keeps it because removal would destroy data you asked it to keep, and Spark has no way to know whether you will read that data again.

One cache freezes the cluster at its peak

Cached blocks live on the executors that ran the tasks computing them. Cache a 400-partition DataFrame while 50 executors are running, and roughly 8 partitions land on each. Every one of the 50 now holds a block, so every one is exempt from removal. Whatever size the cluster had when the cache filled is the size it keeps.
A weekly member-scoring job shows the pattern. It caches active titles, uses them in an early join, runs a 2-hour scoring step that never touches them, and reads them once more for a final report.
titles = spark.read.parquet("/warehouse/titles").filter("is_active").cache()
titles.count()
daily = events.join(titles, "title_id").groupBy("title_id", "event_day").count()
daily.write.mode("overwrite").parquet("/warehouse/daily_title_counts")
scores = score_members(members)
scores.write.mode("overwrite").parquet("/warehouse/member_scores")
report = scores.join(titles, "title_id").groupBy("genre").count()
report.write.mode("overwrite").parquet("/warehouse/genre_report")

The count() fills the cache across all 50 executors. The scoring step's tasks occupy 4 executors for 2 hours, so the target drops to 4 and the other 46 run nothing. With the default timeout all 46 stay, because each holds part of titles. The table prices the 2-hour step at $0.25 per executor-hour under 3 choices.

ChoiceExecutors held during scoringExecutor-hoursCost of the step
Default, timeout infinite50100$25.00
`cachedExecutorIdleTimeout=5min`4, after 5 minutes11.8$2.96
`unpersist()` after the daily write4, after 60 seconds8.8$2.19
92%
idle share of default executor-hours
Both releasing choices pay a recompute: the final report reads titles again and rebuilds the missing partitions from lineage, which here means re-reading one filtered table. Releasing cached executors is cheap when the cached data is cheap to rebuild, and expensive when it sits at the end of hours of joins.
Multiple Choice

A notebook session uses dynamic allocation with every other setting at its default. At 9:00 it caches a 200-partition DataFrame while 40 executors are running, runs 3 queries on it by 9:20, and then sits open with no activity. How many executors does it hold at 15:00?

4 ways to release them

  • unpersist() after the last read that needs the blocks. The 60-second rule applies again at once. Best when the cache has a clear last use.
  • A finite cachedExecutorIdleTimeout. Executors that run no task for that long leave with their blocks, and later reads recompute them. Set it longer than the gap between stages that read the cache.
  • A write and read-back in place of the cache. The data lives in storage, so no executor holds it; a reliable checkpoint() behaves the same, while localCheckpoint() keeps blocks on executors and pins them like a cache.
  • A finite timeout plus spark.decommission.enabled, spark.storage.decommission.enabled and spark.storage.decommission.rddBlocks.enabled: from Spark 3.1, an executor released this way copies its cached blocks to survivors first. Survivors need free storage memory, or the copies evict other blocks.

Locality sends each task that reads a cached partition to the executor holding it, so while stages keep reading the cache, those executors stay busy and neither idle timer fires on them.

With the external shuffle service, spark.shuffle.service.fetch.rdd.enabled=true serves DISK_ONLY cached blocks after their executor leaves, so executors whose blocks sit only on disk follow the 60-second rule. Blocks in memory still pin.

Spot Instances

Daily Life
Interviews
Spot capacity is a provider's spare machines, sold at a discount that providers advertise at up to 90 percent below on-demand prices, on condition that the provider can take them back. Notice runs from 30 seconds to 2 minutes depending on the provider, and then the node is gone with every executor on it and every file on its local disks.

What one reclaim takes

Lost with the nodeWhat Spark doesWhat it costs
Tasks running on itReschedules them on other executorsThe work those tasks had done
Shuffle files it wroteReducers hit fetch failures; Spark reruns the lost map tasks, then retries the stageThose map tasks plus a stage attempt
Its external shuffle serviceNothing to fall back on: the service died with the nodeSame as the shuffle files
Cached blocksRecomputes them from lineage at the next readLineage back to the last durable copy
The driver, if it ran thereThe application endsThe whole run

The shuffle row is where jobs die. Each fetch failure fails the reading stage's attempt, and spark.stage.maxConsecutiveAttempts (4 by default) aborts the job after 4 consecutive failed attempts of one stage. A reclaim wave that takes several nodes over a few minutes can fail the same stage repeatedly, because each retry reruns map tasks on nodes that may go next.

Each map task writes its shuffle output to the executor it ran on. In the join below broadcast is off, so both tables shuffle by title_id, and each of the 3 reducers fetches its share from every map task. Whichever node holds a map task's output is a dependency of every reducer.

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
joined = views.join(titles, "title_id")
joined.collect()
views8 rows
view_id
title_id
member_id
1
10
501
2
20
502
3
10
503
4
30
501
5
40
504
6
20
505
7
30
502
8
10
506
titles4 rows
title_id
genre
10
Drama
20
Comedy
30
Documentary
40
Thriller
Preparing the animation
If the executor that ran one views map task is reclaimed after the map stage, each reducer misses the rows that task wrote. Spark reruns that single map task, then the reducer attempts that failed. Lose 1 node out of 12 and about a twelfth of the map stage reruns; lose 4 at once and a third of it reruns while the reducers wait.

4 decisions for a spot fleet

  • Driver on on-demandNever discount the one process the job cannot loseLosing the driver ends the application. Every other node can come and go.
  • An on-demand floorIts size sets the worst caseDecide how long the job may take if every spot node vanishes, and keep enough on-demand executors to meet that.
  • Several spot poolsMix machine types and zonesReclaims follow demand for one machine type in one zone, so 4 pools of 5 nodes lose 5 at a time where 1 pool of 20 can lose all 20.
  • Smaller nodesFewer executors per reclaimA node with 8 executors takes 8 executors' shuffle files with it; 2 nodes of 4 halve each loss.

Turn the notice into a migration

Graceful decommissioning, available since Spark 3.1, uses the notice. The executor stops accepting tasks and, with storage decommissioning on, copies its shuffle files and cached blocks to other executors, or to a fallback path in durable storage when no peer has room. Reducers then fetch from the new location and no map task reruns. This works only if the cluster manager passes the notice to Spark and the copy finishes inside the window, and an executor holding 30GB of shuffle output may not make it in 30 seconds. The config below also raises maxConsecutiveAttempts to 8, so a reclaim wave gets more attempts before the job aborts.

from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("nightly_rollup")
.config("spark.decommission.enabled", "true")
.config("spark.storage.decommission.enabled", "true")
.config("spark.storage.decommission.shuffleBlocks.enabled", "true")
.config("spark.storage.decommission.rddBlocks.enabled", "true")
.config("spark.storage.decommission.fallbackStorage.path", "file:///mnt/shared/spark-fallback/")
.config("spark.stage.maxConsecutiveAttempts", "8")
.getOrCreate()
)

For a job whose late stages sit on hours of earlier work, a reliable checkpoint() or a write and read-back at the midpoint puts a durable copy outside the cluster. A reclaim after that point recomputes from the copy instead of from the source. Spot lowers the price of each executor-hour and raises the expected rework; durable midpoints cap the rework.

Price the fleet, including the rework

Size the rollup's fleet at 25 executors, where it runs in 28 minutes, against a deadline of 45 minutes. Take on-demand executors at $0.25 an hour and spot at $0.08, and assume reclaims stretch any run with spot in it by 15 percent, to about 32 minutes.
FleetFleet price per hourExpected runtimeExpected costIf every spot node vanishes
25 on-demand$6.2528 min$2.92No effect
15 on-demand + 10 spot$4.5532 min$2.4315 executors finish in 43 min
8 on-demand + 17 spot$3.3632 min$1.798 executors finish in about 69 min
25 spot$2.0032 min$1.07Stalls until spot capacity returns
Every spot fleet is cheaper per run, but the last column tells you which ones you may use. If the 45-minute deadline must hold on the worst night, 15 on-demand executors is the smallest floor that keeps it, and spot can only add speed on top. If the deadline can slip on a bad night, the 8-executor floor saves another 64 cents a run.
Multiple Choice

A 3-hour job runs 40 spot executors on 5 large nodes, with its driver on on-demand and the external shuffle service enabled. In its last hour, reclaims hit 2 nodes a few minutes apart during the final shuffle stage, and the job aborts. Which change most directly keeps it alive?

Batch Scales, Streaming Does Not

Daily Life
Interviews
Dynamic allocation is built for one load shape: work comes in stages whose task counts jump, and between stages or between jobs, executors go quiet for longer than a minute. Batch jobs have that shape. A streaming application has the opposite one: it stays up for days and starts a small job every few seconds as new input arrives, so every executor is touched again and again and none goes quiet.

A stream never trips the idle timer

Take a stream that starts a job every 10 seconds, each finishing in 4 seconds across all executors. No executor ever goes 60 seconds without a task, so none is released. At 8 p.m. input doubles, tasks queue, and the driver requests more executors. Say provisioning takes 2 minutes: that is 12 of the stream's 10-second windows, and the backlog grows while help is on its way. Once the new executors land they receive tasks every 10 seconds like the rest, and they stay through the quiet night. The cluster ratchets up to its largest spike and never comes back down.
✓Nightly batch job
  • Ends after 30 minutes to a few hours and releases everything
  • Work arrives as stages whose task counts are known when each starts
  • Quiet gaps after wide stages let idle executors pass 60 seconds
  • A 2-minute wait for new nodes delays a 28-minute run by under 10 percent
✗Always-on stream
  • Runs for weeks with no end point at which to release resources
  • Work arrives every few seconds and follows the input rate
  • Every executor gets a task in each window, so none reaches 60 seconds idle
  • A 2-minute wait for new nodes spans 12 windows of a 10-second stream

Sizing the fixed cluster a stream needs

Size a stream from its peak. Measure the task-seconds one window needs at the busiest hour, divide by the window length to get the slots that must be busy, and add headroom so a slow window or a restart can be caught up while new input keeps arriving.
StepValue
Peak work in one 10-second window180 task-seconds
Slots busy at peak180 / 10 = 18
With 50% headroom18 x 1.5 = 27 slots
Executors of 4 cores7, giving 28 slots
Time used per window at peak180 / 28 = 6.4 of 10 seconds
The headroom is what clears a backlog. At peak, 28 slots do 28 task-seconds of work each second while 18 arrive, so the cluster works off 10 task-seconds of backlog per second. With no headroom, a backlog formed at peak lasts until input falls.

Deciding per workload

WorkloadLoad shapeClusterStarting settings
Nightly batch with uneven stagesOne 800-task stage, then far fewerDynamicmin 2, max at the deadline size
Shared notebook, ad hoc queriesBursts with long quiet gapsDynamicmin 0, cached timeout 30min
Tight-deadline batch, same every nightPredictable, little slackDynamic with min at the deadline size, or fixedNo ramp wait at the start
Many jobs under a minute eachShort burstsFixed and smallRamp would outlast the work
Iterative training on a cacheSteady, cache-boundFixedCached executors stay anyway
Always-on streamFollows the input rateFixed, peak plus headroomNo dynamic allocation timer fires
TIP
Keep a stream fixed and run its reprocessing and backfills as separate batch applications with dynamic allocation, so their extra capacity leaves when they finish.
Multiple Choice

Which of these workloads gains the most from dynamic allocation, on a cluster that runs the external shuffle service?

Cost per Run

Daily Life
Interviews
Cost per run is executor-hours times the price of one executor-hour, plus the driver's node for the whole runtime. On a fixed cluster, executor-hours is executor count times runtime. Under dynamic allocation it is the area under the executor timeline, which grows during wide stages and shrinks during quiet ones.

Runtime from waves

Each stage runs in waves, ceil(tasks / slots) of them, each as long as one task, so stage time is waves times task duration. Add serial time, the minutes when 1 task or only the driver is working: planning, a single-file write, a final collect(). Runtime is the sum, and on a fixed cluster cost is executors times runtime times price.

Apply it to the nightly rollup: stage A has 800 tasks of 1.5 minutes, stage B has 200 tasks of 3 minutes, and 10 minutes are serial. Executors have 4 cores and cost $0.25 per hour. The deadline is 45 minutes.
ExecutorsSlotsStage AStage BRuntimeCost per runMeets 45 min
104020 waves, 30 min5 waves, 15 min55 min$2.29No
156014 waves, 21 min4 waves, 12 min43 min$2.69Yes
208010 waves, 15 min3 waves, 9 min34 min$2.83Yes
251008 waves, 12 min2 waves, 6 min28 min$2.92Yes
502004 waves, 6 min1 wave, 3 min19 min$3.96Yes
1004002 waves, 3 min1 wave, 3 min16 min$6.67Yes
The cheapest size that meets the deadline is 15 executors. Cost climbs with size for 2 reasons the table shows. The 10 serial minutes are billed at the full cluster, so every executor added pays for them while it waits. Partial waves add the rest: at 15 executors the last wave of stage A runs 20 tasks on 60 slots, and at 100 executors stage B's 200 tasks leave half of 400 slots idle. Work that fills whole waves costs the same at any size, while idle slots grow with it.
$4.17 of $6.67
paid during serial time at 100 executors

What dynamic allocation changes

With dynamic allocation, the external shuffle service, a 60-second idle timeout and minExecutors=1, serial time stops costing the full cluster: 1 executor runs the serial task and the rest leave after a minute. Set maxExecutors to the size, and the bill becomes the parallel executor-minutes plus a small serial tail:

maxExecutorsParallel executor-minutesSerial executor-minutesCost per runRuntime
1549524$2.1643 min
2545034$2.0228 min
5045059$2.1219 min

The cheapest size is now 25, faster than 15 and cheaper. Its 100 slots divide stage A's 800 tasks into exactly 8 waves and stage B's 200 into exactly 2, so no slot waits in a partial wave; at 15 the partial waves cost 45 executor-minutes. Under dynamic allocation, pick maxExecutors so the largest stages divide into whole waves. The model ignores ramp-up, and if the serial step reads stage B's shuffle output under shuffle tracking, all 25 stay through it and the saving disappears. The driver adds its own price per minute: $0.12 on the 28-minute run, $0.18 on the 43-minute one.

Try it on recorded runs

The cell applies the same model to data_pipes, whose dur_secs column records each pipeline run's duration. It assumes every run was measured on 20 executors with 20 percent of its time serial, projects each pipeline's average onto a candidate size, and prices it. Run it at 40, then set candidate_executors to 10: runtime grows and cost falls. At 80, runtime shrinks toward the serial floor while cost grows, because the serial share is billed at every executor.

from pyspark.sql import functions as F

measured_executors = 20
candidate_executors = 40
serial_share = 0.2
price_per_executor_hour = 0.25
sla_secs = 900

time_factor = serial_share + (1 - serial_share) * measured_executors / candidate_executors

runs = (
    spark.table("data_pipes")
    .filter(F.col("pipe_name").isNotNull() & F.col("dur_secs").isNotNull())
    .groupBy("pipe_name")
    .agg(F.avg("dur_secs").alias("measured_secs"))
    .withColumn("candidate_secs", F.round(F.col("measured_secs") * time_factor, 0))
    .withColumn(
        "cost_per_run",
        F.round(F.col("candidate_secs") / 3600 * candidate_executors * price_per_executor_hour, 2),
    )
    .withColumn("meets_sla", F.when(F.col("candidate_secs") <= sla_secs, "yes").otherwise("no"))
    .orderBy(F.desc("cost_per_run"), "pipe_name")
)
runs.show()
Choosing the cluster, step by step
  • Measure 1 run: tasks and median task time for each stage, and the serial minutes.
  • Tabulate runtime and cost for 4 to 6 candidate sizes.
  • Drop every size that misses the deadline once you add margin for ramp-up and retries.
  • Take the cheapest of the rest; under dynamic allocation, set maxExecutors so big stages fill whole waves.
  • Price spot last: fleet price times runtime stretched by your measured rework, against the floor you can accept.
Multiple Choice

The nightly rollup must finish within 45 minutes even if every spot node disappears. On-demand executors cost $0.25 an hour and spot $0.08, and spot in the fleet stretches a run by around 15 percent. Which of these designs meets that at the lowest expected cost?

❯❯❯PUTTING IT ALL TOGETHER

> A video streaming company runs 2 Spark workloads on one cluster manager: an always-on stream that counts playback starts per title, and the nightly rollup that rebuilds viewing history before morning dashboards. Finance asks why last month's compute bill doubled. The job history shows the weekly scoring job holding 50 executors through its 2-hour scoring step, the stream grown to 3 times its usual size after a holiday spike, and the rollup failing twice on spot nodes reclaimed during its final shuffle.

The scoring job kept 50 executors because its early cache pinned them; a finite cachedExecutorIdleTimeout or a write and read-back releases 46 of them.
The stream ratcheted up because no executor ever sat idle for 60 seconds; it moves to a fixed 7-executor cluster, and backfills run as separate elastic applications.
The rollup's failures came from shuffle files lost with reclaimed spot nodes. Priced against its 45-minute deadline, the spot fleet that stays safe saves less than releasing serial time does, so the rollup moves off spot to all on-demand executors.
Under dynamic allocation with maxExecutors at 25, those on-demand executors leave during serial time, so the rollup costs roughly $2.02 a run against $2.43 for the safe spot fleet and finishes in 28 minutes, inside its 45-minute deadline.
KEY TAKEAWAYS
Dynamic allocation targets running plus pending tasks divided by slots per executor, capped by maxExecutors; requests start at 1 and double each round.
Executors leave after 60 idle seconds unless the floor, shuffle files a running job needs, or cached blocks hold them.
cachedExecutorIdleTimeout is infinite by default, so a cache pins the cluster at the size it had when the cache filled.
A spot reclaim destroys the node's shuffle files, its shuffle service and its cached blocks; the driver belongs on on-demand, and the on-demand floor sets the worst case.
A stream touches every executor every few seconds, so dynamic allocation only ratchets it upward; give it a fixed cluster sized to peak plus headroom.
Cost per run is the area under the executor timeline, and serial minutes and partial waves are what make bigger clusters cost more.

Make Spark clusters grow, shrink and survive lost nodes, then price every run against its deadline.

Category
Spark
Difficulty
advanced
Duration
14 minutes
Challenges
13 hands-on challenges

Topics covered: Dynamic Allocation, Scale-Down With Cached Data, Spot Instances, Batch Scales, Streaming Does Not, Cost per Run

Lesson Sections

  1. Dynamic Allocation

    The number the driver targets Ramp-up doubles every round 800 tasks over 4 slots would need 200 executors, so the cap of 50 binds and round 6 adds 17 instead of 32. All 6 requests go out within 6 seconds. Idle nodes start executors within seconds, but a new machine takes a minute or more to arrive after its request. The machines from different rounds provision in parallel, so the ramp waits about one provisioning delay, and stage 1 runs on whatever has landed. Drain-down waits on the idle timer

  2. Scale-Down With Cached Data

    One cache freezes the cluster at its peak Cached blocks live on the executors that ran the tasks computing them. Cache a 400-partition DataFrame while 50 executors are running, and roughly 8 partitions land on each. Every one of the 50 now holds a block, so every one is exempt from removal. Whatever size the cluster had when the cache filled is the size it keeps. A weekly member-scoring job shows the pattern. It caches active titles, uses them in an early join, runs a 2-hour scoring step that ne

  3. Spot Instances

    Spot capacity is a provider's spare machines, sold at a discount that providers advertise at up to 90 percent below on-demand prices, on condition that the provider can take them back. Notice runs from 30 seconds to 2 minutes depending on the provider, and then the node is gone with every executor on it and every file on its local disks. What one reclaim takes If the executor that ran one views map task is reclaimed after the map stage, each reducer misses the rows that task wrote. Spark reruns

  4. Batch Scales, Streaming Does Not

    Dynamic allocation is built for one load shape: work comes in stages whose task counts jump, and between stages or between jobs, executors go quiet for longer than a minute. Batch jobs have that shape. A streaming application has the opposite one: it stays up for days and starts a small job every few seconds as new input arrives, so every executor is touched again and again and none goes quiet. A stream never trips the idle timer Take a stream that starts a job every 10 seconds, each finishing i

  5. Cost per Run

    Cost per run is executor-hours times the price of one executor-hour, plus the driver's node for the whole runtime. On a fixed cluster, executor-hours is executor count times runtime. Under dynamic allocation it is the area under the executor timeline, which grows during wide stages and shrinks during quiet ones. Runtime from waves Apply it to the nightly rollup: stage A has 800 tasks of 1.5 minutes, stage B has 200 tasks of 3 minutes, and 10 minutes are serial. Executors have 4 cores and cost $0