Joins: Advanced
Two ways to bring keys together
- Both sides hashed on the join key and shuffled
- Network cost grows with the size of both tables
- Each worker holds one slice of each side
- Works when both tables are large
- A hot key sends all of its rows to one worker
- The large side never moves
- The small side is copied once per worker
- Each worker holds the entire small side
- Needs one side that fits in memory
- The large side keeps its even spread
Broadcast: each worker keeps the events partitions it already read and receives a full copy of regions, so no events row crosses the network.
The query does not change
Where broadcast breaks
A query joins a 2TB events table to a 40GB accounts table on account_id. The cluster has 100 workers with 8GB of memory each. Which plan runs?
SELECTe.name,e.salary,t.rate AS tax_rateFROM employee_metrics AS eJOIN cost_allocs AS tON e.salary >= t.min_incomeAND e.salary < t.max_income
BETWEEN closes both ends
x BETWEEN a AND b means x >= a AND x <= b, inclusive on both sides. That matters at the seams. If 2 bands touch at 40,000, so that one ends at 40,000 and the next starts at 40,000, an income of exactly 40,000 satisfies both, and that employee comes back twice. Switch between the 3 queries below to see the duplicate appear and then disappear.
A parcel weighs exactly 5.0 kg. The rate table has a band from 0 to 5 and a band from 5 to 10, and the join condition is weight BETWEEN min_kg AND max_kg. How many rows does this parcel produce?
What a range join costs
| ON clause | Comparisons for 1M employees and 300 bands |
|---|---|
| Range only | 300,000,000 |
| Country equality, then range | 3,000,000 with 100 countries and 3 bands each |
SELECTe.name AS employee,m.name AS managerFROM employee_metrics AS eLEFT JOIN employee_metrics AS mON e.manager_id = m.emp_id
Climbing 2 levels
The 6 person employees table is joined to itself with JOIN instead of LEFT JOIN on e.manager_id = m.emp_id. How many rows does SELECT e.name, m.name return?
Counting in the other direction
- ▸Name aliases for their role, e for employee and m for manager, never t1 and t2
- ▸Qualify every column, because name and emp_id exist on both sides of the join
- ▸Decide which side the LEFT JOIN preserves before writing it: the top of the tree or the leaves
Anchor, step, stop
The CTE has 2 members joined by UNION ALL. The anchor runs once and returns the starting rows, here the employees with no manager. The recursive member joins the base table to the CTE by name, so it finds employees whose manager is already in the result. The engine runs the recursive member against the rows the previous pass produced, appends whatever comes back, then runs it again against only those new rows, and stops the first time a pass returns 0 rows.
| Pass | Rows added | depth |
|---|---|---|
| Anchor | Alice | 1 |
| Pass 1 | Bob, Carol | 2 |
| Pass 2 | Dave, Erin, Frank | 3 |
| Pass 3 | none, recursion stops |
The anchor member of a recursive CTE is WHERE manager_id IS NULL, but a data fix set every manager_id to a real value, so no row has NULL. What does the CTE return?
Starting from the middle
When recursion does not stop
- Fixed number of levels, 1 alias each
- All levels for a node in one wide row
- Planned as one set of hash joins
- Any depth, discovered at run time
- One row per node with its depth
- One join per pass, planned pass by pass
A recursive CTE over a 12 level tree runs 12 passes, each one a join against the base table. The engine does not know the depth before it starts, so it cannot plan the whole traversal at once; with a known depth and a large table, a chain of self joins is often faster.
> List Carol and everyone below her in employees, with Carol at depth 0. Order the pieces into the recursive CTE; one piece does not belong.
SELECTa.category,b.device_typeFROM products AS aCROSS JOIN devices AS b
The grid that finds the gaps
The grid has 9 cells for 3 titles and 3 regions. Six are licensed, and the WHERE keeps the 3 whose LEFT JOIN side is NULL: Lunar Bakery in Brazil, and Steel Dawn in both Japan and the United States. Every zero filling report works this way: a grid of days and stores, LEFT JOIN sales, and a day with no sales shows as a row whose sales columns are NULL, which COALESCE(SUM(s.amount), 0) or COUNT(s.sale_id) turns into 0.
Attaching a total to every row
Pairing a table with itself
Accidental products
- Multiply the row counts before running a CROSS JOIN
- Cross join to a 1 row subquery to attach a total
- Pair CROSS JOIN with LEFT JOIN to find missing combinations
- Use a < b to keep each unordered pair once
- Cross join 2 large tables with no filter right after
- Write an ON that is always true and call it an INNER JOIN
- Reach for CROSS JOIN where an equality key exists
- Forget that 1,000 times 1,000 is 1,000,000 rows
A report CROSS JOINs 12,000 titles with 190 regions, LEFT JOINs the availability table, and filters to rows where availability is NULL. How many rows exist before the WHERE runs?
> Netflix's content team wants a weekly report of every title and country pair that lacks a license, rolled up to the top level genre of each title, with the licensing fee band the title's budget falls in and an estimate of the minutes that pair would earn: the title's average weekly minutes per country where it is already licensed. The viewing events side runs to billions of rows; the title, country, genre and fee band tables each hold fewer than 100,000.
Plan joins for clusters, ranges, hierarchies, trees and full grids.
Self joins and CROSS joins. Send help.
- Category
- SQL
- Difficulty
- advanced
- Duration
- 32 minutes
- Challenges
- 15 hands-on challenges
Topics covered: Broadcast joins (distributed), Inequality joins with BETWEEN, Self joins for hierarchical data, Recursive CTEs for trees, CROSS JOIN (Cartesian product)
Lesson Sections
- Broadcast joins (distributed) (concepts: Join Cardinality)
A single machine joins 2 tables by reading both from its own disk. A cluster holds each table in pieces spread across many workers, and a row from the events table can only be paired with a row from the regions table when both sit in the same worker's memory. Every distributed join starts with the same question: which rows have to move so that matching keys meet on one machine? Two ways to bring keys together A partitioned join hashes the join key on both sides and sends each row to the worker t
- Inequality joins with BETWEEN (concepts: Inequality Joins)
Every join so far has paired rows on equality: a key on the left equals a key on the right. The ON clause accepts any condition that returns true, false or NULL for a pair of rows, and a range condition pairs a row with the band its value falls inside. Tax rates by income, shipping prices by weight, surge multipliers by hour and discounts by order size are all bands with a lower bound and an upper bound, stored as rows. The cost_allocs table holds the tax bands: each row is a min_income, a max_i
- Self joins for hierarchical data (concepts: Self-Join)
An employees table stores each person's manager as a manager_id that points back into the same table. Asking for each employee's manager's name is therefore a join between employees and employees. The database does not care that both sides are the same table: it treats each alias as an independent copy, and the row pairing model from the beginner lesson applies unchanged. The alias e stands for the employee side and m for the manager side, and the ON clause reads: the employee's manager_id equal
- Recursive CTEs for trees (concepts: Recursive Queries)
A chain of self joins reaches as many levels as it has aliases: 2 aliases, 2 levels. A genre catalog nested 9 deep, or an org chart that runs 4 levels in one division and 11 in another, needs a query that keeps joining until there is nothing left to join. A recursive CTE does that. It names a result set, seeds it with starting rows, and keeps adding rows derived from the rows it already holds until a pass adds none. Anchor, step, stop The depth column is carried by hand: the anchor sets it to 1
- CROSS JOIN (Cartesian product) (concepts: Cross Joins)
A CROSS JOIN has no ON clause and pairs every row on the left with every row on the right. Two tables of 2 rows give 4 pairs; 12,000 titles and 190 regions give 2,280,000. Nothing is matched or filtered, so the product is the full set of combinations, which is what you want whenever the question is about combinations that should exist rather than rows that already do. Every category meets every device type. The result has 2 times 2 rows, and a third device would make it 6. Think of CROSS JOIN as