BeginnerSpark · 12 min · 14 challenges

Data Skew: Beginner

Watch a slow Spark job in the UI and you will often see the same strange shape: the stage sprints to 199 of 200 tasks in half a minute, then sits there. One task grinds on alone for another 40 minutes while a cluster you are paying for by the second does nothing. Nothing is broken. No error will ever appear. The data itself is lopsided: one partition got a pile of rows far bigger than everyone else's, and the whole job is hostage to the one task chewing through it. That lopsidedness is called data skew, and it is one of the most common reasons real production jobs are slow. This tier teaches you to recognize it: what skew is, the straggler shape it makes, where it comes from, how to confirm it in the Spark UI, and why it wastes the entire cluster.

What Skew Is

Daily Life

Interviews

You can now define data skew precisely: partitions of wildly unequal size, minted by a shuffle that groups rows by key.

Everything Spark does is built on an assumption so quiet you may never have said it out loud: that the data is divided evenly. A job's data is split into partitions, one task processes each partition, and the tasks run in parallel across the cluster. When every partition holds roughly the same number of rows, every task does roughly the same amount of work and finishes at roughly the same time. That is the happy case all the parallelism math assumes. Data skew is the name for the case where the assumption fails: one partition, or a few, hold far more data than their peers.

The imbalance is not random bad luck in how files were read. When Spark reads a folder of similar-sized Parquet files, the input partitions come out reasonably even, because they are cut by bytes, not by meaning. The place skew gets minted is the shuffle. You know from the narrow and wide lesson that a wide operation regroups rows by key so that every row for a given key lands in the same partition. That regrouping is exactly the mechanism that concentrates data: the partition sizes after a shuffle are decided by how many rows each key has, and real keys are never even.

Be precise about what is unequal. Every post-shuffle partition gets roughly the same number of keys, because the hash function spreads keys uniformly. What it cannot spread is the rows behind a key. If customer_id has a million customers with a hundred rows each and one customer with 50 million rows, the partition that draws that one customer holds 50 million rows plus its ordinary share, while every other partition holds its ordinary share alone. The hash did its job perfectly; the data was lopsided before the hash ever saw it.

A property of the data, not the code

This is why skew is worth its own lesson: nothing in your code announces it. The same groupBy that runs evenly on one table skews badly on another, and the same pipeline that ran fine for a year can start skewing the week a single customer's traffic explodes. Skew is a property of the data's distribution, so you cannot find it by reading the code, and adding hardware does not remove it. A job with a skewed partition is slow on 10 executors and just as slow on 100, because the problem is concentrated in one place.

✓Balanced partitions (the assumption)

Every partition holds a similar share of rows
Every task does similar work
Tasks finish together; the stage ends
Doubling executors roughly halves the time

⚠Skewed partitions (the reality)

One partition holds a huge share of rows
One task does most of the work
The stage waits for its biggest partition
Extra executors just wait too

Hold onto the working definition: skew is when the size gap between the biggest partition and the typical partition is large enough to matter, and it usually appears right after a shuffle, because the shuffle is what groups rows by key. A rough rule of thumb is that a biggest partition a few times the median is normal life, while 10 times or more is skew worth acting on. The next section shows what that gap looks like from the outside, because you will almost always meet skew as a symptom in a running job before you ever see a partition size.

TIP

When someone says a table 'has skew', ask 'on which key?'. A table is not skewed by itself; a distribution of rows per key is. The same orders table can be perfectly even when grouped by order date and brutally skewed when grouped by customer, because skew belongs to the key you shuffle on, not to the data as a whole.

The Straggler Task

Daily Life

Interviews

You can now recognize a straggler task as the signature symptom of skew, and know what it is doing to your schedule.

Skew announces itself with one unmistakable shape: the straggler. A stage launches 200 tasks. In the first half minute, 199 of them finish. The last one keeps running, for 10 minutes, for 45, sometimes for hours, while the stage progress bar sits one tick from complete. Nothing errors. Nothing retries. The job is not stuck in any way a health check can see; it is simply one task working through a partition many times bigger than everyone else's. If you spend time around production Spark, you will see this shape so often it becomes reflex to name it before you have opened a single tab.

The reason one task can hold a whole stage is the barrier you met in the shuffle lesson. A stage does not end until every one of its tasks ends, because the next stage needs all of the regrouped data in place before it can start. So the wall-clock time of a stage is not the average task time; it is the maximum. 199 tasks finishing in 30 seconds contribute nothing to the finish line if the two-hundredth needs 45 minutes. The slowest task is the stage, and under skew the slowest task is the one that drew the hot partition.

199 in 30s, 1 in 45min

the straggler shape

This is also why progress percentages lie to you under skew. A stage showing 99.5 percent complete sounds nearly done, but tasks are not equal slices of work; they are equal counts of partitions. If the one remaining task holds a quarter of the stage's data, the stage is really 75 percent done by rows and may have most of its runtime still ahead. Reading task counts as progress is exactly the mistake skew punishes. The honest measure of how far along a skewed stage is would be bytes processed, and the UI's per-task metrics, which a later section walks through, are how you approximate it.

What the straggler costs you

The daily-life consequence is scheduling pain. Skewed stragglers are the classic reason a nightly job blows its window: the pipeline that comfortably finished by 6am creeps to 6:40, then 7:30, as one customer or one product grows, and downstream dashboards and SLAs start missing with it. It is a 3am page with no error in the logs, which makes it confusing the first time: every task succeeded, the job completed, and it was still 2 hours late. When the only symptom is lateness and the stage timeline shows one long bar at the end, you are looking at skew.

One caution before you blame the data: a single slow task has two possible causes, a big partition or a slow machine. A dying disk, a noisy neighbor, or a garbage-collection storm can make a normal-sized partition crawl too. The two look identical from the progress bar and completely different in the fix, which is why the diagnosis in the UI section leans on data volume per task, not just duration. For now, keep the shape itself: many tasks fast, one task endless, no failures. That is the straggler, and the rest of this lesson is about tracing it to its cause.

Where Hot Keys Come From

Daily Life

Interviews

You can now predict which keys in your own tables will run hot before you ever run the job.

Behind almost every straggler is a hot key: one value of the grouping or join column that owns a wildly outsized share of the rows. Real-world data is not uniform, it is top-heavy. One enterprise customer generates a third of your events. One country dominates your user base. One SKU is the product everyone buys. One referrer sends most of your traffic. When you shuffle on that column, every row carrying the hot value is delivered to the same partition, because landing same-key rows together is the entire point of a shuffle. The mechanism doing its job correctly is what creates the imbalance.

This top-heaviness has a name: power-law, or 80/20, distributions. Measure almost any human-generated key and you find a few values with enormous counts and a long tail of values with tiny ones. Word frequencies, city populations, purchases per customer, and events per device all follow the same curve. Uniformly distributed keys are the rare, artificial case: synthetic IDs and well-designed surrogate keys behave; anything that encodes real behavior or geography usually does not. Which is why skew is not an exotic corner case in Spark; it is the default condition of shuffling business data by a meaningful column.

Key you shuffle on	The likely hot value	Why it is hot
customer_id	Your biggest enterprise account	A few whales generate most of the events
country / region	The home market	One geography dominates the user base
product_id	The bestseller, or a default SKU	Hits follow a power law
event_type	page_view or heartbeat	High-volume telemetry drowns rare events
any nullable column	NULL itself	Every missing value collects into one bucket

Two hot keys deserve special mention because they are manufactured, not natural. Placeholder values are the first: an unknown or default id like -1, 0, or UNKNOWN that an upstream system stamps on every row it could not resolve. 10 different failures upstream all funnel into the same placeholder, quietly building the biggest key in the table. The second is null, which behaves like a placeholder that nobody chose: every row missing the column carries the same nothing, and all of that nothing shuffles to the same place. The intermediate tier gives null skew a full section, because it hides in otherwise healthy pipelines.

Reading your own schema for heat

The practical skill is prediction. Before you run a groupBy or a join, look at the key and ask what its biggest value's share probably is. Grouping by a date usually behaves, because a day's traffic is bounded. Grouping by user id in a consumer app usually behaves, until you remember the test account or the scraper bot. Joining on customer id at a B2B company almost never behaves, because revenue concentration is the business model. You know your own domain's whales; skew is where they show up in the runtime. Naming the risky key before the job runs is the cheapest diagnosis you will ever do.

Notice what this section did not say: nothing about your code being wrong. A hot key skews a perfectly written groupBy, because the concentration lives in the data. That is also why skew follows data growth. The pipeline was fine when your biggest customer was 2 percent of traffic; three years of that customer growing faster than the rest, and the same code straggles every night. When an old, untouched job starts slowing down, the question is not what changed in the code. It is which key's distribution drifted underneath it.

Spotting Skew in the UI

Daily Life

Interviews

You can now confirm skew in the Spark UI by reading a stage's max-versus-median task metrics.

The Spark UI settles the question of whether a slow stage is skewed, and it does it with one comparison: the maximum task against the median task. Open the application UI, go to the Stages tab, and sort by duration to find the stage eating the time. Click into that stage and find the Summary Metrics table. It shows the distribution of every task-level number across the stage, min, 25th percentile, median, 75th percentile, and max, for duration, input size, shuffle read, and more. That little table is the skew detector.

Read the duration row first. In a healthy stage, max duration sits within a small multiple of the median: the slowest task took maybe twice or three times the middle one, which is ordinary variance. In a skewed stage the ratio is dramatic: a median of 30 seconds against a max of 45 minutes is a ratio of 90. There is no hard threshold where variance officially becomes skew, but a useful working rule is that a max around 10 times the median deserves an explanation, and a max 100 times the median is a diagnosis.

Duration alone is not proof, because a slow machine also produces a long max. The confirmation is the data volume rows: shuffle read size and records for a post-shuffle stage, input size for a scan. If the max-duration task also shows a max shuffle read many times the median, the task was slow because it was fed more data, and that is skew. If the durations are lopsided but the bytes are even, the data was balanced and something environmental, a bad node or a GC storm, made one task crawl. Same symptom, opposite fixes, and the two columns side by side tell them apart.

Summary metric	Healthy stage	Skewed stage
Duration: max vs median	Within 2-3x	10x or more
Shuffle read: max vs median	Roughly even	One task reads many times the median
Records: max vs median	Roughly even	One task processes most of the rows
Failures	None	Often none; sometimes the max task OOMs

The tasks table below the summary lets you go one step further and look at the straggler itself. Sort tasks by duration and the top row is your suspect: note its shuffle read, its record count, and which executor it ran on. A skewed task shows the oversized read; if instead the same executor keeps hosting slow tasks across different stages while its data volumes look normal, suspicion moves back to the machine. One more tell is worth knowing: when the straggler eventually fails with an out-of-memory error and retries, over and over, that is skew graduating from a slowness problem to a stability problem, because the one partition no longer fits in one task's memory.

From symptom to key

The UI tells you which stage is skewed and how badly, but it does not tell you which key is responsible; it counts bytes, not values. Getting from the straggling stage to the guilty key means querying the data itself, which is where the intermediate tier begins. What you take from this tier is the reading habit: slow stage, open summary metrics, compare max to median on duration, then confirm with shuffle read. 30 seconds in the UI replaces an afternoon of guessing, and it is the difference between saying the job feels slow and saying partition 137 got 50 times the data.

TIP

The summary metrics table is also your defense against fixing the wrong problem. Before touching a slow stage, note its max and median. After your fix, compare. If the max fell to a small multiple of the median, you fixed the skew; if the whole distribution shifted down instead, you sped the stage up but the imbalance, and the straggler, will be back as the data grows.

Why Skew Wastes the Cluster

Daily Life

Interviews

You can now explain why one hot partition idles every other executor, runs up the bill, and resists more hardware.

It is tempting to file one slow task under annoyance rather than emergency: the job still finishes, after all. The reason skew deserves the emergency file is arithmetic. While the straggler grinds, every other core in the cluster is idle, and you are paying for all of them. Take a modest cluster of 50 executors with 4 cores each, 200 cores total. The stage's 199 normal tasks finish in 30 seconds; the straggler runs another 44 minutes. For those 44 minutes, 199 of your 200 cores do nothing. That is close to 150 core-hours of paid, allocated, idle compute burned on a single stage, every time it runs.

The bill is one half of the waste; the schedule is the other. Because a shuffle boundary is a barrier, the stage after the straggler cannot start until it lands, so the delay propagates through every downstream stage, every downstream job, and every human waiting on the output. Skew turns a parallel system into a serial one at its worst possible point: the whole pipeline advances at the pace of one core. Utilization charts make it vivid: a cluster that was 95 percent busy during the healthy part of the stage collapses to under 1 percent busy for the straggler's long tail.

1 of 200 cores working

what the cluster does during a straggler

Now the trap: the reflexive fix, add more executors, does nothing. Scaling out helps when work is divisible; the straggler's work is one partition, and a partition is processed by exactly one task on one core. Doubling the cluster to 400 cores gets the healthy tasks done marginally sooner and then parks 399 cores instead of 199 while the same straggler runs the same 44 minutes on the same single core. You made the job more expensive and no faster. When someone reaches for a bigger cluster to cure a straggler, the arithmetic above is the two-line explanation of why the bill went up and the runtime did not move.

There is a stability cost hiding here too. The hot partition concentrates not just time but memory on one executor: its task must hold and process vastly more data than its peers, so it spills to disk first, and at some growth point it becomes the task that starts failing with out-of-memory errors while the rest of the job looks healthy. Skew is usually met as slowness, but its endgame is a job that cannot finish at all, failing on the same task every retry. Catching it while it is merely slow is much cheaper than meeting it as an outage.

The shape of the fix

Everything in this tier was recognition, and recognition points directly at the shape a real fix must have. The problem is that one key's rows are indivisible under the shuffle's normal rules, so the fix has to change the rules: split the hot key's rows so they can spread across many tasks, or route around the hot key entirely. Making that happen, measuring the exact keys, salting them apart, isolating them, and the special case of null keys, is the intermediate tier. What you carry forward from here: straggler shape, max versus median, hot key, idle cluster. You can now spot skew; next you fix it.

Multiple Choice

A stage runs 200 tasks: 199 finish in 40 seconds, one runs for an hour, and the UI shows that task read 60x the median shuffle read. Your teammate proposes doubling the executor count. What happens?

✓Do

Read a stage's max task against its median before proposing any fix.
Confirm skew with data volume (shuffle read, records), not duration alone.
Translate straggler time into idle core-hours when you argue for fixing it.
Watch old jobs for creeping stragglers; skew arrives with data drift, not code changes.

✗Don't

Don't trust a stage's task-count progress bar; tasks are equal counts, not equal work.
Don't add executors to cure a straggler; one partition still runs on one core.
Don't assume a slow task means skew before checking its input size; slow nodes mimic it.
Don't ignore a straggler that still finishes; its endgame is an OOM that never does.

❯❯❯PUTTING IT ALL TOGETHER

> The nightly revenue rollup that has finished by 6am for a year starts landing at 7:30, and this morning it missed its SLA outright. There are no errors in the logs; every run eventually succeeds. The only change anyone can name is that the company signed its largest-ever customer last quarter.

The clean logs plus growing lateness fit skew's profile: nothing fails, one task just takes longer each week.

In the UI you find the aggregation stage where 199 tasks finish in a minute and one runs for 90; its shuffle read is 70x the median, confirming data, not a bad node.

The rollup groups by customer_id, and the new whale customer is exactly the kind of hot key that concentrates rows into one partition.

You decline the offer of a bigger cluster, because the hot partition would still run on one core, and take the diagnosis into the intermediate tier's fixes: measure the keys, then split or bypass them.

KEY TAKEAWAYS

Data skew is a lopsided distribution of rows per key that leaves one partition far bigger than its peers.

The symptom is the straggler: nearly all tasks finish fast while one runs on, and the stage waits for it.

Skew comes from hot keys: whales, home markets, placeholders, and nulls that own an outsized share of rows.

The Spark UI proves it: max task duration and shuffle read many times the median means skew, not a slow node.

One hot partition idles the rest of the cluster and resists more hardware; the fix must split or bypass the hot key.

One partition gets the pile. The whole cluster waits on it.

Category: Spark
Difficulty: beginner
Duration: 12 minutes
Challenges: 14 hands-on challenges

Topics covered: What Skew Is, The Straggler Task, Where Hot Keys Come From, Spotting Skew in the UI, Why Skew Wastes the Cluster

Lesson Sections

What Skew Is (concepts: paDataSkew)
Everything Spark does is built on an assumption so quiet you may never have said it out loud: that the data is divided evenly. A job's data is split into partitions, one task processes each partition, and the tasks run in parallel across the cluster. When every partition holds roughly the same number of rows, every task does roughly the same amount of work and finishes at roughly the same time. That is the happy case all the parallelism math assumes. Data skew is the name for the case where the
The Straggler Task (concepts: paDataSkew)
Skew announces itself with one unmistakable shape: the straggler. A stage launches 200 tasks. In the first half minute, 199 of them finish. The last one keeps running, for 10 minutes, for 45, sometimes for hours, while the stage progress bar sits one tick from complete. Nothing errors. Nothing retries. The job is not stuck in any way a health check can see; it is simply one task working through a partition many times bigger than everyone else's. If you spend time around production Spark, you wil
Where Hot Keys Come From (concepts: paDataSkew)
Behind almost every straggler is a hot key: one value of the grouping or join column that owns a wildly outsized share of the rows. Real-world data is not uniform, it is top-heavy. One enterprise customer generates a third of your events. One country dominates your user base. One SKU is the product everyone buys. One referrer sends most of your traffic. When you shuffle on that column, every row carrying the hot value is delivered to the same partition, because landing same-key rows together is
Spotting Skew in the UI (concepts: paDataSkew)
The Spark UI settles the question of whether a slow stage is skewed, and it does it with one comparison: the maximum task against the median task. Open the application UI, go to the Stages tab, and sort by duration to find the stage eating the time. Click into that stage and find the Summary Metrics table. It shows the distribution of every task-level number across the stage, min, 25th percentile, median, 75th percentile, and max, for duration, input size, shuffle read, and more. That little tab
Why Skew Wastes the Cluster (concepts: paDataSkew)
It is tempting to file one slow task under annoyance rather than emergency: the job still finishes, after all. The reason skew deserves the emergency file is arithmetic. While the straggler grinds, every other core in the cluster is idle, and you are paying for all of them. Take a modest cluster of 50 executors with 4 cores each, 200 cores total. The stage's 199 normal tasks finish in 30 seconds; the straggler runs another 44 minutes. For those 44 minutes, 199 of your 200 cores do nothing. That