Broadcasting the Small Side
Concepts covered: Broadcast Join
Move the small table, leave the big one
In the example, products has 3 rows and orders has 6. Shuffling orders to meet products costs the same as shuffling orders for any other purpose: every row written to disk, every row read back over the network. There is another way to get the matching rows into one task. Leave every orders partition where it is, and send the entire products table to every executor. Each task that holds an orders partition then has all 3 products in front of it and finds each order's product with a local hash lookup.
That is a broadcast join. The driver collects the small table, builds a hash table keyed on product_id, and sends 1 copy to each executor. The big table's partitions never move, so the join runs as a narrow transformation over orders, inside the same stage as whatever transformation came before it. The 2 shuffles of the previous section are gone, and so is the sort: a hash lookup needs no sorted input.
F.broadcast(products) is a hint that tells Spark to broadcast that side whatever it estimates the size to be. Watch the animation below: products leaves its partitions once, arrives whole at every orders task, and the output appears without any row of orders changing executor.
orders.join(F.broadcast(products), "product_id").collect()| order_id | product_id |
|---|---|
| 1 | 4017 |
| 2 | 2210 |
| 3 | 4017 |
| 4 | 3305 |
| 5 | 2210 |
| 6 | 4017 |
| product_id | product_name |
|---|---|
| 4017 | Desk lamp |
| 2210 | Notebook |
| 3305 | Headphones |
What the broadcast costs instead
A broadcast has its own bill. The small table travels once to each executor, and each executor keeps a hash table of it in memory for the length of the stage. A 20MB dimension on a cluster of 50 executors therefore crosses the network 50 times, and each of the 50 executors holds 20MB more than it otherwise would. Set that against shuffling a 200GB fact table, where every byte is written to disk and read back, and the broadcast is the smaller number by a factor of a few hundred.
- 1GB
- 400GB
When Spark broadcasts without being asked
You do not always need the hint. Before choosing a strategy, Spark estimates the size of each side. If one side's estimate is under spark.sql.autoBroadcastJoinThreshold, which defaults to 10MB, Spark broadcasts that side on its own. That is what the -1 in section 0 switched off: setting the threshold to -1 tells Spark that no table is small enough, so both sides shuffle. The estimate comes from the files Spark reads, and after a filter or an aggregation it can be far from the truth, so those are the joins where you add the hint yourself.
events is 200GB and countries is 5MB, both read from files with default settings. How many shuffles does events.join(countries, "country_code") add to the job?
- 1, because events is hashed on country_code to meet the broadcast copy
- 2, because a join always hashes both sides on the key before matching them
- 0, because countries is under the threshold and is broadcast to every executor
- 1, because countries is shuffled to the partitions that hold the events rows
Run the same fact-to-dimension join as before, now with the hint on products. The rows that come back are identical; what changed is where the work happened. order_items stayed in its partitions and products travelled to them.
About this lesson
This section is part of the Joins: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.