Cost per Run
Cost per run is executor-hours times the price of one executor-hour, plus the driver's node for the whole runtime. On a fixed cluster, executor-hours is executor count times runtime. Under dynamic allocation it is the area under the executor timeline, which grows during wide stages and shrinks during quiet ones.
Runtime from waves
Each stage runs in waves, ceil(tasks / slots) of them, each as long as one task, so stage time is waves times task duration. Add serial time, the minutes when 1 task or only the driver is working: planning, a single-file write, a final collect(). Runtime is the sum, and on a fixed cluster cost is executors times runtime times price.
Apply it to the nightly rollup: stage A has 800 tasks of 1.5 minutes, stage B has 200 tasks of 3 minutes, and 10 minutes are serial. Executors have 4 cores and cost $0.25 per hour. The deadline is 45 minutes.
| Executors | Slots | Stage A | Stage B | Runtime | Cost per run | Meets 45 min |
|---|---|---|---|---|---|---|
| 10 | 40 | 20 waves, 30 min | 5 waves, 15 min | 55 min | $2.29 | No |
| 15 | 60 | 14 waves, 21 min | 4 waves, 12 min | 43 min | $2.69 | Yes |
| 20 | 80 | 10 waves, 15 min | 3 waves, 9 min | 34 min | $2.83 | Yes |
| 25 | 100 | 8 waves, 12 min | 2 waves, 6 min | 28 min | $2.92 | Yes |
| 50 | 200 | 4 waves, 6 min | 1 wave, 3 min | 19 min | $3.96 | Yes |
| 100 | 400 | 2 waves, 3 min | 1 wave, 3 min | 16 min | $6.67 | Yes |
The cheapest size that meets the deadline is 15 executors. Cost climbs with size for 2 reasons the table shows. The 10 serial minutes are billed at the full cluster, so every executor added pays for them while it waits. Partial waves add the rest: at 15 executors the last wave of stage A runs 20 tasks on 60 slots, and at 100 executors stage B's 200 tasks leave half of 400 slots idle. Work that fills whole waves costs the same at any size, while idle slots grow with it.
- $4.17 of $6.67
What dynamic allocation changes
With dynamic allocation, the external shuffle service, a 60-second idle timeout and minExecutors=1, serial time stops costing the full cluster: 1 executor runs the serial task and the rest leave after a minute. Set maxExecutors to the size, and the bill becomes the parallel executor-minutes plus a small serial tail:
| maxExecutors | Parallel executor-minutes | Serial executor-minutes | Cost per run | Runtime |
|---|---|---|---|---|
| 15 | 495 | 24 | $2.16 | 43 min |
| 25 | 450 | 34 | $2.02 | 28 min |
| 50 | 450 | 59 | $2.12 | 19 min |
The cheapest size is now 25, faster than 15 and cheaper. Its 100 slots divide stage A's 800 tasks into exactly 8 waves and stage B's 200 into exactly 2, so no slot waits in a partial wave; at 15 the partial waves cost 45 executor-minutes. Under dynamic allocation, pick maxExecutors so the largest stages divide into whole waves. The model ignores ramp-up, and if the serial step reads stage B's shuffle output under shuffle tracking, all 25 stay through it and the saving disappears. The driver adds its own price per minute: $0.12 on the 28-minute run, $0.18 on the 43-minute one.
Try it on recorded runs
The cell applies the same model to data_pipes, whose dur_secs column records each pipeline run's duration. It assumes every run was measured on 20 executors with 20 percent of its time serial, projects each pipeline's average onto a candidate size, and prices it. Run it at 40, then set candidate_executors to 10: runtime grows and cost falls. At 80, runtime shrinks toward the serial floor while cost grows, because the serial share is billed at every executor.
Choosing the cluster, step by step
- Measure 1 run: tasks and median task time for each stage, and the serial minutes.
- Tabulate runtime and cost for 4 to 6 candidate sizes.
- Drop every size that misses the deadline once you add margin for ramp-up and retries.
- Take the cheapest of the rest; under dynamic allocation, set maxExecutors so big stages fill whole waves.
- Price spot last: fleet price times runtime stretched by your measured rework, against the floor you can accept.
The nightly rollup must finish within 45 minutes even if every spot node disappears. On-demand executors cost $0.25 an hour and spot $0.08, and spot in the fleet stretches a run by around 15 percent. Which of these designs meets that at the lowest expected cost?
- Run 25 spot executors and an on-demand driver: $1.07 a run, the lowest price of the four
- Run 8 on-demand plus 17 spot: $1.79 a run, with an on-demand floor that keeps the job alive
- Run 15 on-demand plus 10 spot on a fixed cluster: $2.43 a run, and the 15 alone finish in 43 minutes
- Run 25 on-demand under dynamic allocation with maxExecutors 25: about $2.02 a run, with no spot to lose
About this lesson
This section is part of the Cluster Sizing: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.