How Do You Size the Cluster?

Concepts: paCostOptimization, paMemoryManagement

Cluster sizing becomes cost modeling. The question shifts from 'how many executors' to 'how do you minimize $/query while meeting the SLA?' Dynamic allocation, spot instances, cluster pooling, and chargeback models are the dimensions. Dynamic Allocation in Practice Dynamic allocation requests executors when tasks are pending and releases them after an idle timeout (default 60s). The scaling is reactive, not predictive - there is a 30-60 second lag between demand and allocation because YARN/K8s must provision containers. For jobs with bursty shuffle stages, this lag means the cluster is undersized during peaks and oversized during valleys. Spot Instance Strategy Spot instances are 60-90% cheaper than on-demand but can be reclaimed with 2 minutes notice. The strategy: run the driver and a