Batch Scales, Streaming Does Not
Dynamic allocation is built for one load shape: work comes in stages whose task counts jump, and between stages or between jobs, executors go quiet for longer than a minute. Batch jobs have that shape. A streaming application has the opposite one: it stays up for days and starts a small job every few seconds as new input arrives, so every executor is touched again and again and none goes quiet.
A stream never trips the idle timer
Take a stream that starts a job every 10 seconds, each finishing in 4 seconds across all executors. No executor ever goes 60 seconds without a task, so none is released. At 8 p.m. input doubles, tasks queue, and the driver requests more executors. Say provisioning takes 2 minutes: that is 12 of the stream's 10-second windows, and the backlog grows while help is on its way. Once the new executors land they receive tasks every 10 seconds like the rest, and they stay through the quiet night. The cluster ratchets up to its largest spike and never comes back down.
- Nightly batch job
- Ends after 30 minutes to a few hours and releases everything
- Work arrives as stages whose task counts are known when each starts
- Quiet gaps after wide stages let idle executors pass 60 seconds
- A 2-minute wait for new nodes delays a 28-minute run by under 10 percent
- Always-on stream
- Runs for weeks with no end point at which to release resources
- Work arrives every few seconds and follows the input rate
- Every executor gets a task in each window, so none reaches 60 seconds idle
- A 2-minute wait for new nodes spans 12 windows of a 10-second stream
Sizing the fixed cluster a stream needs
Size a stream from its peak. Measure the task-seconds one window needs at the busiest hour, divide by the window length to get the slots that must be busy, and add headroom so a slow window or a restart can be caught up while new input keeps arriving.
| Step | Value |
|---|---|
| Peak work in one 10-second window | 180 task-seconds |
| Slots busy at peak | 180 / 10 = 18 |
| With 50% headroom | 18 x 1.5 = 27 slots |
| Executors of 4 cores | 7, giving 28 slots |
| Time used per window at peak | 180 / 28 = 6.4 of 10 seconds |
The headroom is what clears a backlog. At peak, 28 slots do 28 task-seconds of work each second while 18 arrive, so the cluster works off 10 task-seconds of backlog per second. With no headroom, a backlog formed at peak lasts until input falls.
Deciding per workload
| Workload | Load shape | Cluster | Starting settings |
|---|---|---|---|
| Nightly batch with uneven stages | One 800-task stage, then far fewer | Dynamic | min 2, max at the deadline size |
| Shared notebook, ad hoc queries | Bursts with long quiet gaps | Dynamic | min 0, cached timeout 30min |
| Tight-deadline batch, same every night | Predictable, little slack | Dynamic with min at the deadline size, or fixed | No ramp wait at the start |
| Many jobs under a minute each | Short bursts | Fixed and small | Ramp would outlast the work |
| Iterative training on a cache | Steady, cache-bound | Fixed | Cached executors stay anyway |
| Always-on stream | Follows the input rate | Fixed, peak plus headroom | No dynamic allocation timer fires |
Which of these workloads gains the most from dynamic allocation, on a cluster that runs the external shuffle service?
- A stream that processes payment events every 5 seconds, around the clock
- A 3-minute job that runs every 5 minutes all day on the same data
- A nightly backfill whose first stage has 2,000 tasks and whose last writes 16 files over 40 minutes
- A training job that caches its input once and then reads it in 50 passes
About this lesson
This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.