This Job Is Slow

Concepts: paSparkExecutionModel, paMemoryManagement

The debugging question escalates from 'use the Spark UI' to 'walk me through your systematic methodology when the Spark UI isn't enough.' Flame graphs, GC analysis, and spill forensics are the tools that separate debugging from guessing. The Systematic Methodology GC Analysis Spark defaults to G1GC for executors. When the old generation fills, G1 triggers full GC pauses that halt all task execution. A healthy executor spends < 5% of time in GC. Above 10%, you are memory-starved. Above 20%, the executor is effectively stalled and tasks will start timing out. Spill Analysis Spill happens when execution memory (shuffles, sorts, aggregations) exceeds the allocated fraction and Spark writes intermediate data to local disk. Disk spill is 10-100x slower than memory. The Spark UI shows 'Shuffle Sp