Which Line Costs the Most

Concepts covered: paShuffleOptimization

The natural drill is to ask you to rank your own operations: which line costs the most. The junior move is to identify the single widest transform, the one that moves the most data. In the example, the groupBy is the dominant cost, because it shuffles the full filtered dataset by key, whereas the final orderBy shuffles only the already-aggregated rows, which are far fewer. Reasoning that the earlier, larger shuffle dominates the later, smaller one shows you understand that not all shuffles cost the same. What they are fishing for What the interviewer is fishing for is that position in the pipeline changes a shuffle's cost. The same operation, a sort, is cheap or expensive depending on how much data has already been reduced before it. Here the groupBy collapses millions of order rows into a

About This Interactive Section

This section is part of the SQL at Scale lesson on DataDriven, a free data engineering interview prep platform. Each section includes explanations, worked examples, and hands-on code challenges that execute in real time. SQL queries run against a live database. Python runs in a sandboxed Docker container. Data modeling problems validate against interactive schema canvases. All content is framed around what data engineering interviewers actually test at companies like Meta, Google, Amazon, Netflix, Stripe, and Databricks.

How DataDriven Lessons Work

DataDriven combines four interview rounds (SQL, Python, Data Modeling, Pipeline Architecture) with adaptive difficulty and spaced repetition. Easy problems get harder as you improve. Weak concepts resurface until you master them. Your readiness score tracks progress across every topic interviewers test. Every lesson section ends with problems you solve by writing and running real code, not by picking multiple-choice answers.