# A daily Spark join runs for hours

Canonical URL: <https://datadriven.io/problems/a-daily-spark-join-runs-for-hours-the-spark-ui-shows-one-ta-249970e2>

Domain: Pipeline Design · Difficulty: medium

## Problem

A daily Spark join runs for hours. The Spark UI shows one task running for 90% of the runtime while the rest finish fast: a single hot key is skewing the shuffle. Adding executors didn't help. Design the job so the skewed join completes on time.

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/a-daily-spark-join-runs-for-hours-the-spark-ui-shows-one-ta-249970e2)
- [System Design Interview Questions](https://datadriven.io/data-engineering-system-design)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). 100% free data engineering interview prep. Live code execution against Postgres 16, Python 3.11, and Spark sandboxes. No paywall, no premium tier, no signup gate.