Broadcasting the Small Side

Concepts covered: paBroadcastJoin

Look at the orders-to-products join again with fresh eyes. Orders is 400 million rows; products is 2 million rows, maybe 80 megabytes on disk. The default plan shuffles both, which means 400 million order rows cross the network to meet a table that would fit in the memory of a phone. That is backwards. If products is small enough to hold in memory, the cheaper move is obvious: leave the giant table exactly where it is, and send a complete copy of the small table to every executor instead. That is a broadcast join, and its mechanics are worth having precisely. The small table is collected into one place, serialised, and shipped whole to every executor in the cluster. Each executor builds it into an in-memory hash map keyed by the join key. Then the big table is joined without moving at all:

About This Interactive Section

This section is part of the Joins: Beginner lesson on DataDriven, a free data engineering interview prep platform. Each section includes explanations, worked examples, and hands-on code challenges that execute in real time. SQL queries run against a live database. Python runs in a sandboxed Docker container. Data modeling problems validate against interactive schema canvases. All content is framed around what data engineering interviewers actually test at companies like Meta, Google, Amazon, Netflix, Stripe, and Databricks.

How DataDriven Lessons Work

DataDriven combines four interview rounds (SQL, Python, Data Modeling, Pipeline Architecture) with adaptive difficulty and spaced repetition. Easy problems get harder as you improve. Weak concepts resurface until you master them. Your readiness score tracks progress across every topic interviewers test. Every lesson section ends with problems you solve by writing and running real code, not by picking multiple-choice answers.