Pricing a Shuffle
Every shuffle sends a bill with 4 lines, and each line scales with a number on the stage page. Pricing a shuffle means reading those numbers and turning them into disk traffic, network traffic and request counts before deciding what to change.
| Line | Work | Paid by | Scales with |
|---|---|---|---|
| Write | serialize, compress, write; reread and rewrite spills | map tasks | Shuffle Write Size, Spill (Disk) |
| Serve | read byte ranges back off disk | the node holding the files | Shuffle Write Size and block count |
| Network | move blocks between nodes | the cluster network | remote share of Shuffle Read Size |
| Read | decompress, deserialize, aggregate or sort | reduce tasks | Shuffle Read Size / Records |
Shuffle output is compressed by default: spark.shuffle.compress is true and the codec is lz4. Shuffle Write Size therefore reports compressed bytes, the same bytes that later cross the disk and the network.
A worked price
A daily job reads 600GB of clickstream, keeps 4 columns, and groups by user. The map stage reports a Shuffle Write Size of 90GB across 2,400 tasks. The reduce stage runs 600 tasks spread over 60 executors, so a reducer finds about 1 in 60 of its blocks on its own executor.
- 1.44M
| Figure | Arithmetic |
|---|---|
| Disk traffic | 90GB written + 90GB read back = 180GB, before any spill |
| Network | 90GB × 59/60 = 88.5GB |
| Blocks | 2,400 map tasks × 600 reducers = 1,440,000 |
| Average block | 90GB ÷ 1,440,000 = 62.5KB |
| Per reducer | 2,400 fetches totalling 150MB |
The bytes look large, but the block size is the line that warns you. At 62.5KB per block, each reducer makes 2,400 small reads to collect 150MB. Fast local SSDs absorb that, while spinning disks spend more time seeking than transferring. Doubling the reducers to 1,200 halves the average block to about 31KB and doubles the total fetches to 2.88 million. Each reducer still makes 2,400 reads, now for 75MB. Shuffle Write Size stays at 90GB.
A map stage reports a Shuffle Write Size of 40GB from 1,000 tasks. The reduce stage runs 500 tasks across 50 executors. Which estimate fits?
- About 39.2GB crosses the network, in blocks averaging 40MB
- About 0.8GB crosses the network, in blocks averaging 80KB
- About 80GB crosses the network, once to write and once to read, in 80KB blocks
- About 39.2GB crosses the network, in blocks averaging 80KB
Shrink the rows before you pay
Every line of the bill multiplies rows crossing the boundary by bytes per row. A filter and a narrow select before the wide operation cut both, and the partial aggregate cuts rows to 1 per key per map task. The board plays a filter and a sum over 6 orders read as 2 partitions and shuffled to 2 reducers.
(orders
.filter(F.col("status") == "completed")
.groupBy("customer_id")
.agg(F.sum("amount").alias("spend"))
.collect())| order_id | customer_id | status | amount |
|---|---|---|---|
| 1 | c1 | completed | 40 |
| 2 | c2 | cancelled | 25 |
| 3 | c1 | completed | 60 |
| 4 | c3 | completed | 35 |
| 5 | c2 | completed | 50 |
| 6 | c1 | cancelled | 20 |
Of the 6 orders, 4 survive the filter, and each map task sends 1 partial sum for every customer it holds, so the cancelled orders never reach the shuffle. The driver collects c1 at 100, c2 at 50 and c3 at 35.
Counting what a shuffle would carry
- Shuffle Write Size / Records on the map stage: the compressed bytes and the rows the map tasks wrote.
- Shuffle Read Size / Records on the reduce stage, with Shuffle Remote Reads giving the network line.
- Shuffle Read Blocked Time: how long reducers sat waiting for fetches instead of computing.
- Spill (Memory) and Spill (Disk): rows a writer or reader sorted in more than 1 pass.
The cell below reads 3 counts that bound a shuffle of the seed's orders: rows read, rows left after keeping completed orders, and the distinct regions among them. For a groupBy("region") on completed orders, that last count caps the partial sums each map task can send: at most 1 per distinct region it holds, plus 1 for rows whose region is null, since countDistinct skips nulls. Run it, then change "Completed" to another status and see which counts move.
About this lesson
This section is part of the Shuffles: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.