# Size the Executors > Too big: GC kills you. Too small: broadcast kills you. Canonical URL: Domain: Spark · Difficulty: medium · Seniority: L5 · Asked in: Microsoft ## Problem On a 50-node cluster where every machine has 128 GB and 32 cores, keeps 1 core and 1 GB for the OS, and splits its remaining memory in whole gigabytes among as many executors as its free cores allow, each executor gives up 2 GB of that share to off-heap overhead and keeps the rest as heap. The recommendations team's feature job already died on 16-core executors from GC thrash and on 4 GB executors that could not hold its 2.1 GB broadcast table, so only 1 to 5 cores per executor are in play; with no cluster table to read, number the rows of `users` and take positions 1 through 5 as those core counts. List the sizings whose heap is at least three times the broadcast size, with cores per executor, executors per node, heap in GB, and the executor count across the whole cluster. ## Expected output First 3 of 3 rows: | cores_per_executor | executors_per_node | heap_gb | total_executors | | --- | --- | --- | --- | | 3 | 10 | 10 | 500 | | 4 | 7 | 16 | 350 | | 5 | 6 | 19 | 300 | ## Worked solution and explanation ### What this really is This is a two-sided constraint problem wearing an incident costume. The skill being probed: can you turn cores per executor into heap size by doing the node arithmetic honestly, then hold each sizing against a memory floor that pulls the opposite way from the thread cap? Anyone can say "use 5 cores". The trap is the memory you never get: the gigabyte the OS keeps and the 2 GB each executor hands to overhead. Skip them and the 2-core sizing looks like an 8 GB heap that clears the broadcast floor, when it really has 6 GB and dies exactly like Config B. > **Heap is a function of cores, not a knob** > > On a fixed node you do not pick memory and cores independently. Fewer cores per executor means more executors per node, which means a thinner slice of RAM each. Enumerate the core count and let everything else fall out. #### Step 1: Number the `users` rows to get candidate core counts The cap of 5 threads per JVM comes straight from Config A, so the only candidates are 1 through 5. Numbering `users` in `user_id` order with `ROW_NUMBER()` gives a gap-free counter whatever the ids look like; keep positions up to 5 and call it `cores_per_executor`. #### Step 2: Take the OS reserve off, then let cores decide executors Only 31 cores and 127 GB per node are schedulable. `executors_per_node` is `FLOOR(31 / cores_per_executor)`, and leftover cores sit idle: 4-core executors fit 7 per node and strand 3 cores, which is the honest price of that sizing. #### Step 3: Split memory, then pay overhead Each executor's container is `FLOOR(127 / executors_per_node)` whole gigabytes, and 2 GB of that is off-heap overhead, so `heap_gb` is the container minus 2. YARN counts the overhead against the node; your tasks just cannot use it. #### Step 4: Apply the broadcast floor Config B says the 2.1 GB broadcast needs room beside execution and storage memory, so the heap must be at least three times it: `heap_gb >= 6.3`. That floor cuts the 1-core and 2-core sizings and leaves three. **Enumerate and filter the sizings** ```python BROADCAST_GB = 2.1 by_user = Window.orderBy("user_id") candidates = ( spark.table("users") .select(F.row_number().over(by_user).alias("cores_per_executor")) .where(F.col("cores_per_executor") <= 5) .withColumn("executors_per_node", F.floor(F.lit(31) / F.col("cores_per_executor"))) ) sized = candidates.withColumn( "heap_gb", F.floor(F.lit(127) / F.col("executors_per_node")) - 2 ) result = ( sized .where(F.col("heap_gb") >= 3 * BROADCAST_GB) .select( "cores_per_executor", "executors_per_node", "heap_gb", (F.lit(50) * F.col("executors_per_node")).alias("total_executors"), ) ) ``` *Three sizings survive: 3, 4 and 5 cores, with 10, 16 and 19 GB heaps.* **Naive memory split** 2 cores: 16 executors per node, 128 / 16 = 8 GB heap. Clears the 6.3 GB floor, so it ships. **Honest memory split** 2 cores: `FLOOR(31 / 2)` = 15 per node, `FLOOR(127 / 15)` = 8 GB container, minus 2 GB overhead = 6 GB heap. Fails the floor. > **The broadcast lives once per executor** > > A broadcast table is materialized in every executor JVM, not every task. That is exactly why tiny 1-core executors are wasteful: 31 copies of 2.1 GB per node instead of 6, and each copy squeezed into a 2 GB heap. > **Pick one survivor and defend it** > > Listing the feasible band is half the answer. The senior move is choosing 5 cores and 19 GB (300 executors, 1,500 cores) for heap headroom on the broadcast join, or 4 cores and 16 GB (350 executors, 1,400 cores) for shorter GC pauses, and saying why. > **Big heaps cost pause time** > > Full collections scale with live heap. A 64 GB heap shared by 16 threads spent over 80% of its time collecting; at 16 to 19 GB with 4 or 5 threads, GC dropped from roughly 1,840 seconds to about 20 per executor. > **The cluster manager takes its cut too** > > On YARN the application master also needs a container, so production configs usually request one executor fewer than the arithmetic allows. The sizing logic is unchanged; only the instance count shifts. ## Common follow-up questions - The broadcast table grows to 6 GB. Which sizings survive, and when do you stop broadcasting? _(Tests whether the floor moves with the data and when a shuffle join becomes cheaper.)_ - Nodes move to 256 GB and 64 cores. Does the 5-core cap still hold? _(Separates the per-JVM thread limit from the node shape.)_ - How would dynamic allocation change `total_executors` for a job with a long skewed tail? _(Probes idle executor release and straggler stages.)_ ## Related - [All practice problems](https://datadriven.io/problems) - [Mock interview mode](https://datadriven.io/interview/spark_executor_memory_config) - [PySpark Interview Questions](https://datadriven.io/pyspark-interview-questions) - [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep) - [Daily Challenge](https://datadriven.io/daily) --- Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.