Hash vs Range Partitioning
Concepts covered: Shuffle Optimization
Every map task has to send each row to exactly 1 reducer, using a rule every other map task also applies. For shuffles keyed on columns, Spark has 2 rules. Hash partitioning sends a row to partition pmod(hash(key), n), which hashes the key columns with Murmur3, divides the hash by the partition count n, and keeps the non-negative remainder. Range partitioning cuts the key's sorted values into n contiguous intervals and sends each row to the interval its key falls in. A third rule, round-robin, deals rows out in turn with no key at all; repartition(n) uses it.
- Hash partitioning
- Used by groupBy, distinct, joins and repartition(col)
- Same key, same partition, for a fixed n
- Neighbouring keys scatter across partitions
- Needs no pass over the data before the shuffle
- Sizes even out when keys are many and similar in size
- Range partitioning
- Used by orderBy, sort and repartitionByRange(col)
- Same key, same partition, within 1 run
- Every key in partition i sorts before every key in partition i+1
- Runs a sampling job first to pick boundaries
- Sizes depend on how well the sample matches the data
Why a sort needs ranges
A groupBy reducer only needs every row for its keys, and a hash delivers that. A global sort needs more. Each reducer sorts its own rows, and reading partition 0, then 1, then 2 has to yield 1 ordered result, which holds only if every key in partition 0 sorts before every key in partition 1. A hash sends adjacent values like 99 and 100 to unrelated partitions, so orderBy uses ranges.
To place the boundaries, Spark first samples the data: around 100 keys per output partition by default, the value of spark.sql.execution.rangeExchange.sampleSizePerPartition, drawn from every input partition. It sorts the sample and picks n minus 1 cut points that split it evenly. That sampling is a job of its own, the kind you saw run ahead of an action's main job, and it computes the input once for the sample and again for the shuffle.
Suppose the sample for 3 partitions yields cut points of 50 and 100. Rows then route like this:
| amount | Range partition | Why |
|---|---|---|
| 30 | 0 | 50 or less |
| 45 | 0 | 50 or less |
| 60 | 1 | above 50, 100 or less |
| 95 | 1 | above 50, 100 or less |
| 120 | 2 | above 100 |
The animation sorts 9 orders by amount into 3 reduce partitions, with cut points drawn from its own sample. Watch which amounts reach each reducer: low ones at the first, high ones at the last, and each reducer sorts only its own share. Collected in partition order, the result runs 15, 30, 45, 60, 70, 80, 95, 120, 140.
orders.orderBy("amount").collect()| order_id | amount |
|---|---|
| 1 | 120 |
| 2 | 80 |
| 3 | 45 |
| 4 | 60 |
| 5 | 30 |
| 6 | 95 |
| 7 | 15 |
| 8 | 140 |
| 9 | 70 |
Which rule your code gets
| Code | Partitioning | Shuffles |
|---|---|---|
| `df.groupBy("region").count()` | hash on region | 1 |
| `df.distinct()` | hash on every column | 1 |
| `df.orderBy("amount")` | range on amount | 1 |
| `df.repartition(50, "region")` | hash on region, 50 partitions | 1 |
| `df.repartitionByRange(50, "amount")` | range on amount, 50 partitions | 1 |
| `df.repartition(50)` | round-robin, 50 partitions | 1 |
| `df.groupBy("region").count().orderBy("count")` | hash on region, then range on count | 2 |
The partition count is part of the hash
Because the rule ends in a remainder, the same key lands in unrelated partitions under different counts: US might sit in partition 41 of 200 and partition 7 of 50. Two outputs hash-partitioned on the same column line up partition by partition only when they used the same n. Range layouts line up even less often, since their boundaries come from a sample of 1 run's data.
Neither rule splits a key: every row with country = 'US' lands in 1 partition under hash and under range alike.
A job runs events.repartitionByRange(4, "event_date") over a year of events. Which is true of the 4 output partitions?
- Each holds about a quarter of every month's dates, so all 4 cover the whole year
- Each holds 1 contiguous span of dates, and no single date is split between 2 partitions
- Each holds 1 calendar quarter, since range boundaries come from the column's type
- Each holds a scattered set of dates, chosen so the partitions come out equal in size
About this lesson
This section is part of the Shuffles: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.