Cost per Run

Cost per run is executor-hours times the price of one executor-hour, plus the driver's node for the whole runtime. On a fixed cluster, executor-hours is executor count times runtime. Under dynamic allocation it is the area under the executor timeline, which grows during wide stages and shrinks during quiet ones.

Runtime from waves

Each stage runs in waves, ceil(tasks / slots) of them, each as long as one task, so stage time is waves times task duration. Add serial time, the minutes when 1 task or only the driver is working: planning, a single-file write, a final collect(). Runtime is the sum, and on a fixed cluster cost is executors times runtime times price.

Apply it to the nightly rollup: stage A has 800 tasks of 1.5 minutes, stage B has 200 tasks of 3 minutes, and 10 minutes are serial. Executors have 4 cores and cost $0.25 per hour. The deadline is 45 minutes.

ExecutorsSlotsStage AStage BRuntimeCost per runMeets 45 min
104020 waves, 30 min5 waves, 15 min55 min$2.29No
156014 waves, 21 min4 waves, 12 min43 min$2.69Yes
208010 waves, 15 min3 waves, 9 min34 min$2.83Yes
251008 waves, 12 min2 waves, 6 min28 min$2.92Yes
502004 waves, 6 min1 wave, 3 min19 min$3.96Yes
1004002 waves, 3 min1 wave, 3 min16 min$6.67Yes

The cheapest size that meets the deadline is 15 executors. Cost climbs with size for 2 reasons the table shows. The 10 serial minutes are billed at the full cluster, so every executor added pays for them while it waits. Partial waves add the rest: at 15 executors the last wave of stage A runs 20 tasks on 60 slots, and at 100 executors stage B's 200 tasks leave half of 400 slots idle. Work that fills whole waves costs the same at any size, while idle slots grow with it.

What dynamic allocation changes

With dynamic allocation, the external shuffle service, a 60-second idle timeout and minExecutors=1, serial time stops costing the full cluster: 1 executor runs the serial task and the rest leave after a minute. Set maxExecutors to the size, and the bill becomes the parallel executor-minutes plus a small serial tail:

maxExecutorsParallel executor-minutesSerial executor-minutesCost per runRuntime
1549524$2.1643 min
2545034$2.0228 min
5045059$2.1219 min

The cheapest size is now 25, faster than 15 and cheaper. Its 100 slots divide stage A's 800 tasks into exactly 8 waves and stage B's 200 into exactly 2, so no slot waits in a partial wave; at 15 the partial waves cost 45 executor-minutes. Under dynamic allocation, pick maxExecutors so the largest stages divide into whole waves. The model ignores ramp-up, and if the serial step reads stage B's shuffle output under shuffle tracking, all 25 stay through it and the saving disappears. The driver adds its own price per minute: $0.12 on the 28-minute run, $0.18 on the 43-minute one.

Try it on recorded runs

The cell applies the same model to data_pipes, whose dur_secs column records each pipeline run's duration. It assumes every run was measured on 20 executors with 20 percent of its time serial, projects each pipeline's average onto a candidate size, and prices it. Run it at 40, then set candidate_executors to 10: runtime grows and cost falls. At 80, runtime shrinks toward the serial floor while cost grows, because the serial share is billed at every executor.

Choosing the cluster, step by step

The nightly rollup must finish within 45 minutes even if every spot node disappears. On-demand executors cost $0.25 an hour and spot $0.08, and spot in the fleet stretches a run by around 15 percent. Which of these designs meets that at the lowest expected cost?

About this lesson

This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.