Batch Scales, Streaming Does Not

Dynamic allocation is built for one load shape: work comes in stages whose task counts jump, and between stages or between jobs, executors go quiet for longer than a minute. Batch jobs have that shape. A streaming application has the opposite one: it stays up for days and starts a small job every few seconds as new input arrives, so every executor is touched again and again and none goes quiet.

A stream never trips the idle timer

Take a stream that starts a job every 10 seconds, each finishing in 4 seconds across all executors. No executor ever goes 60 seconds without a task, so none is released. At 8 p.m. input doubles, tasks queue, and the driver requests more executors. Say provisioning takes 2 minutes: that is 12 of the stream's 10-second windows, and the backlog grows while help is on its way. Once the new executors land they receive tasks every 10 seconds like the rest, and they stay through the quiet night. The cluster ratchets up to its largest spike and never comes back down.

Nightly batch job
  • Ends after 30 minutes to a few hours and releases everything
  • Work arrives as stages whose task counts are known when each starts
  • Quiet gaps after wide stages let idle executors pass 60 seconds
  • A 2-minute wait for new nodes delays a 28-minute run by under 10 percent
Always-on stream
  • Runs for weeks with no end point at which to release resources
  • Work arrives every few seconds and follows the input rate
  • Every executor gets a task in each window, so none reaches 60 seconds idle
  • A 2-minute wait for new nodes spans 12 windows of a 10-second stream

Sizing the fixed cluster a stream needs

Size a stream from its peak. Measure the task-seconds one window needs at the busiest hour, divide by the window length to get the slots that must be busy, and add headroom so a slow window or a restart can be caught up while new input keeps arriving.

StepValue
Peak work in one 10-second window180 task-seconds
Slots busy at peak180 / 10 = 18
With 50% headroom18 x 1.5 = 27 slots
Executors of 4 cores7, giving 28 slots
Time used per window at peak180 / 28 = 6.4 of 10 seconds

The headroom is what clears a backlog. At peak, 28 slots do 28 task-seconds of work each second while 18 arrive, so the cluster works off 10 task-seconds of backlog per second. With no headroom, a backlog formed at peak lasts until input falls.

Deciding per workload

WorkloadLoad shapeClusterStarting settings
Nightly batch with uneven stagesOne 800-task stage, then far fewerDynamicmin 2, max at the deadline size
Shared notebook, ad hoc queriesBursts with long quiet gapsDynamicmin 0, cached timeout 30min
Tight-deadline batch, same every nightPredictable, little slackDynamic with min at the deadline size, or fixedNo ramp wait at the start
Many jobs under a minute eachShort burstsFixed and smallRamp would outlast the work
Iterative training on a cacheSteady, cache-boundFixedCached executors stay anyway
Always-on streamFollows the input rateFixed, peak plus headroomNo dynamic allocation timer fires

Which of these workloads gains the most from dynamic allocation, on a cluster that runs the external shuffle service?

About this lesson

This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.