This Job Is Slow

Concepts covered: paSparkUiDiagnosis

"This Spark job used to take 20 minutes. Now it takes 3 hours. What do you do?" This is the most common Spark question across all companies. The interviewer is testing your debugging methodology, not a single trick. The Debugging Sequence Check shuffle metrics FIRST. 80% of slow Spark jobs are shuffle-bound. Open the Spark UI → Stages tab → sort by shuffle write. If one stage writes 500GB of shuffle data while others write 5GB, you found the problem. Do not start with code review - start with the UI. Reading the Spark UI The Stages tab shows each stage's task count, duration, shuffle read/write, and input/output size. Click into a stage to see the task-level distribution. A healthy stage has uniform task durations. A sick stage has one task at 45 minutes and 199 tasks at 30 seconds - t

About This Interactive Section

This section is part of the The Spark Deep Dive: Intermediate lesson on DataDriven, a free data engineering interview prep platform. Each section includes explanations, worked examples, and hands-on code challenges that execute in real time. SQL queries run against a live database. Python runs in a sandboxed Docker container. Data modeling problems validate against interactive schema canvases. All content is framed around what data engineering interviewers actually test at companies like Meta, Google, Amazon, Netflix, Stripe, and Databricks.

How DataDriven Lessons Work

DataDriven combines four interview rounds (SQL, Python, Data Modeling, Pipeline Architecture) with adaptive difficulty and spaced repetition. Easy problems get harder as you improve. Weak concepts resurface until you master them. Your readiness score tracks progress across every topic interviewers test. Every lesson section ends with problems you solve by writing and running real code, not by picking multiple-choice answers.