What a Partition Is

Concepts covered: Distributed Primitives

A partition is the unit of work in Spark: a slice of a DataFrame that one task reads and transforms on one core, with no help from any other task. On disk, a partition is a byte range: a whole file, or a piece of a file, with a start offset and a length. In memory, it's the block of rows that one executor holds for that slice. Every narrow transformation runs partition by partition, so a filter keeps the exact count it was given, and so does a select or a withColumn.

Two numbers describe any partition: how many bytes it holds, and how many rows that becomes once decoded. The first decides how long its task spends reading. The second decides how much executor memory the task needs while it works. Both come back in the sections that follow, because every partitioning problem is one of those 2 numbers sitting far from where it should be.

Where partitions come from on read

When you call spark.read on a directory, Spark lists the files and cuts them into partitions of one split size. spark.sql.files.maxPartitionBytes caps the split at 128MB by default. The split shrinks when a read is small so that every core still gets work: Spark divides the total bytes, plus a 4MB charge per file, by the core count and takes that when it's smaller, with 4MB as the floor. Any read above 128MB times the core count splits at 128MB: a 10GB file on 16 cores becomes 80 partitions, read by 80 tasks in parallel. Files larger than the split are cut at its boundaries, and smaller files are packed together up to the split.

The count depends only on file sizes and the core count, so you can predict it from a directory listing and the cluster size before a single row is read. That matters because the count is also the number of tasks the first stage launches, and nothing in a narrow transformation changes it.

orders.filter(F.col("status") == "refunded").collect()

The program above reads a 12-row orders table as 4 partitions and filters it down to the 2 refunded rows. Follow the task count: 4 tasks launch, each scans its 3 rows, and 4 result sets travel back to the driver even though at least 2 of them carry nothing. The filter removed 83% of the rows and left the partition count alone.

Why 128MB

Each task carries a fixed cost: the driver serializes and schedules it, an executor opens the input, and the result is reported back, a few milliseconds of overhead whatever the size of the input. Reading and decoding 128MB takes a task on the order of a second or more, so at that size the overhead stays under 1% of the work. Memory sets the upper limit. A task decodes its partition into rows in memory, and the decoded rows take more space than the bytes on disk, so a 128MB input expands into something that still fits inside the memory one task gets, where a 2GB input often does not. At 128MB a read also leaves enough partitions to spread the work: a 10GB read gives 80 tasks, which keep a 16-core cluster busy for 5 rounds, so one slow task shifts the finish line by a fraction of a round rather than doubling it.

• 2 to 3
• 200ms

Both figures come from Spark's own tuning guide, the first so that every core always has a queue of work waiting, the second as the floor below which scheduling dominates. A partition of 128MB on splittable input lands well above that floor and well below the point where a single task outgrows its memory.

A directory holds one 2GB CSV file and nothing else. On a 16-core cluster, you call spark.read.option("header", "true").csv(path) with default settings. How many partitions does the first stage read, and what does each hold?

• 1 partition, because a single file is read by a single task
• 16 partitions, each a 128MB byte range of the file
• 200 partitions, the default count for any DataFrame
• 2 partitions of 1GB, one per core of a 2-core driver

About this lesson

This section is part of the Partitioning: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.