Spotting Skew in the UI

Concepts covered: paDataSkew

The Spark UI settles the question of whether a slow stage is skewed, and it does it with one comparison: the maximum task against the median task. Open the application UI, go to the Stages tab, and sort by duration to find the stage eating the time. Click into that stage and find the Summary Metrics table. It shows the distribution of every task-level number across the stage, min, 25th percentile, median, 75th percentile, and max, for duration, input size, shuffle read, and more. That little table is the skew detector. Read the duration row first. In a healthy stage, max duration sits within a small multiple of the median: the slowest task took maybe twice or three times the middle one, which is ordinary variance. In a skewed stage the ratio is dramatic: a median of 30 seconds against a ma

About This Interactive Section

This section is part of the Data Skew: Beginner lesson on DataDriven, a free data engineering interview prep platform. Each section includes explanations, worked examples, and hands-on code challenges that execute in real time. SQL queries run against a live database. Python runs in a sandboxed Docker container. Data modeling problems validate against interactive schema canvases. All content is framed around what data engineering interviewers actually test at companies like Meta, Google, Amazon, Netflix, Stripe, and Databricks.

How DataDriven Lessons Work

DataDriven combines four interview rounds (SQL, Python, Data Modeling, Pipeline Architecture) with adaptive difficulty and spaced repetition. Easy problems get harder as you improve. Weak concepts resurface until you master them. Your readiness score tracks progress across every topic interviewers test. Every lesson section ends with problems you solve by writing and running real code, not by picking multiple-choice answers.