Cores and Slots

An executor is a JVM with a fixed number of cores, and each core runs 1 task at a time. Spark calls that capacity a slot: an executor with 4 cores has 4 slots, and 20 such executors give the application 80 slots, so at most 80 tasks run at any instant no matter how many the plan contains. A task occupies its slot from the moment it is scheduled until it reports back, and the next waiting task takes the slot the instant it frees.

Two settings fix the number. spark.executor.cores sets the cores per executor, and on YARN and Kubernetes it defaults to 1, so an application that never sets it runs 1 task per executor however large the machine. spark.executor.instances sets how many executors the driver asks for. Slots are the product of the 2, and the cluster manager can grant fewer executors than requested, which shrinks the product without any change to the code.

Tasks run in waves

When a plan has more tasks than slots, the tasks run in waves. A 400 GB Parquet read at 128 MB per partition is 3,200 partitions and so 3,200 tasks. On 80 slots that is 40 waves, and if each task takes 15 seconds the read takes 40 times 15, 600 seconds, 10 minutes. Doubling the executors to 40 halves the waves to 20 and the time to 5 minutes, because each wave still takes 15 seconds. The same 3,200 tasks on 1 executor with 4 cores would take 800 waves and over 3 hours. Nothing about the data changed between those 3 runs; only the slot count did.

The last wave is where slots go to waste. With 3,210 tasks instead of 3,200, wave 41 runs 10 tasks and leaves 70 slots idle for a full 15 seconds. That costs as much as a whole wave, for only 10 tasks' worth of rows. A task count that is an exact multiple of the slot count keeps the final wave full, and a count a few tasks over a multiple pays almost a wave for nothing. Uneven tasks hurt the same way. If 1 task in a wave takes 90 seconds, its 79 neighbours finish at 15 and their slots pick up the next wave, but the stage cannot end before that 1 task does. In practice slots free one by one and waves blur into each other, and total task seconds divided by slots still sets the floor on elapsed time.

Memory is shared by the cores

spark.executor.memory is per executor, and every core in that executor draws from the same heap. Spark sets aside about 300 MB and gives roughly 60% of the rest, set by spark.memory.fraction, to a pool that running tasks share for processing and caching, so a 16 GB heap leaves about 9.4 GB in that pool. With 4 cores, each running task works in roughly 2 GB of it, and the same 16 GB with 8 cores halves that to about 1 GB. A 128 MB Parquet partition can expand to several hundred megabytes of decoded rows in memory, and a task that groups or sorts them needs working space on top. Adding cores to an executor without adding memory is how a job that ran clean at 4 cores starts spilling to disk at 8. The slot count sets how many tasks run at once, and the memory per slot sets whether each of them finishes without spilling.

A read produces 240 tasks of about 30 seconds each. The application has 10 executors with 8 cores each, all granted. Roughly how long does the read take?

• About 90 seconds
• About 30 seconds
• About 2 hours
• About 4 minutes

With processes, partitions, laziness and slots in hand, the last section follows 1 program through all of them, from the line that calls the action to the rows that reach the driver.

About this lesson

This section is part of the Execution Model: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.