IntermediatePipeline Architecture · 35 min

The Spark Deep Dive: Intermediate

Spark is the single most interrogated technology in data engineering interviews. The pattern is consistent: explain the architecture, debug a slow job, optimize a join, size the cluster, handle skew. Five moves. If you can execute all five cleanly, you pass the technical gauntlet. If you fumble the shuffle explanation or blank on broadcast thresholds, the loop ends early.

What you will be able to do

Draw the Spark execution model from driver to task without hesitation

Diagnose a slow job using shuffle metrics and the Spark UI

Choose the right join strategy and explain the tradeoffs

Size executors, cores, and memory for a given workload

Detect and resolve data skew with salting, repartitioning, and isolation

Explain Spark Architecture

Daily Life

Interviews

Draw the Spark execution model from driver to task

"Walk me through how Spark executes a query." This is the opener. If your answer is vague, the interviewer downgrades your level. If it is precise, they trust your debugging answers later.

The Driver-Executor Model

plans + schedules

driver

5 cores

executor 1

5 cores

executor 2

shuffle exchange

shuffle

Parquet

output

Spark's driver-executor model: the driver builds the DAG and schedules tasks; executors run them in parallel (5-core rule); a wide transformation forces a shuffle exchange (the stage boundary) before the result is written.

Spark runs one driver process and N executor processes. The driver parses your code into a logical plan, optimizes it via Catalyst, converts it to a physical plan, and splits that plan into stages. Each stage contains tasks - one task per partition. The driver sends tasks to executors, which run them in parallel on their allocated cores.

Component	Role	Key Resource
Driver	Plans, schedules, coordinates	Memory for broadcast vars + result collection
Executor	Runs tasks, caches data	Cores × memory per executor
Task	Unit of work on one partition	1 core slot, partition-sized memory

Stages and Shuffle Boundaries

Spark splits the physical plan into stages at shuffle boundaries. A shuffle happens whenever data must be redistributed across the cluster - groupBy, join, repartition, distinct. Everything before a shuffle can be pipelined within a single stage. Everything after requires waiting for the shuffle to complete.

•Narrow Transformations

map, filter, select, withColumn
Each output partition depends on ONE input partition
Pipelined within a stage - no data movement
Essentially free at scale

•Wide Transformations

groupBy, join, repartition, distinct
Output partitions depend on ALL input partitions
Trigger a shuffle - data crosses the network
Dominant cost in most Spark jobs

The DAG (Directed Acyclic Graph) is the full execution plan. Each node is an RDD or DataFrame operation. Edges represent dependencies. Narrow dependencies chain into one stage. Wide dependencies create new stages. The Spark UI shows this DAG visually - learn to read it.

TIP

In the interview, volunteer that you think in terms of stages, not individual transformations. Say: 'I look at the DAG to count shuffle boundaries - that tells me the shape of the job.' This signals depth.

When the interviewer asks why your job was slow, partitioning is often the answer, and citing a number shows you know what you're talking about. The default spark.sql.shuffle.partitions is 200, which is too low for terabyte-scale data and too high for small datasets. Say: 'I set shuffle partitions to data_size / 128MB. For a 500GB job, that's about 4,000 partitions, enough parallelism without drowning the driver in scheduler overhead.'

The Number to Know

▸Target partition size: ~128MB. Below 64MB you have too many small tasks. Above 256MB you risk OOM and poor parallelism.
▸For a 100GB dataset: 100GB / 128MB ≈ 800 partitions is a reasonable starting point.

KEY TAKEAWAYS

Driver plans and schedules; executors run tasks on partitions

Stages split at shuffle boundaries - wide transformations create new stages

The DAG is your debugging map; count the shuffles to understand cost

Target 128MB per partition; adjust spark.sql.shuffle.partitions accordingly

This Job Is Slow

Daily Life

Interviews

Debug a slow Spark job using Spark UI metrics

"This Spark job used to take 20 minutes. Now it takes 3 hours. What do you do?" This is the most common Spark question across all companies. The interviewer is testing your debugging methodology, not a single trick.

The Debugging Sequence

Check shuffle metrics FIRST. 80% of slow Spark jobs are shuffle-bound. Open the Spark UI → Stages tab → sort by shuffle write. If one stage writes 500GB of shuffle data while others write 5GB, you found the problem. Do not start with code review - start with the UI.

Reading the Spark UI

The Stages tab shows each stage's task count, duration, shuffle read/write, and input/output size. Click into a stage to see the task-level distribution. A healthy stage has uniform task durations. A sick stage has one task at 45 minutes and 199 tasks at 30 seconds - that is skew.

Spark UI Metric	Healthy	Investigate
Max task time / Median	< 3x	> 10x (skew)
GC time / Task time	< 5%	> 10% (memory)
Shuffle spill (disk)	0 bytes	> 0 (memory pressure)
Shuffle write	Proportional to data	> 10x input (cartesian?)
Peak exec memory	< 60% of alloc	> 80% (OOM risk)

Shuffle spill means Spark ran out of execution memory and wrote intermediate data to disk. Any spill is a red flag. It means your partitions are too large, your executor memory is too low, or both. The fix is usually repartitioning to smaller partitions or increasing spark.executor.memory.

TIP

When the interviewer asks 'this job is slow,' say: 'My first move is the Spark UI stages tab, sorted by duration, looking at shuffle metrics and task distribution.' Starting with the UI instead of guessing at code changes is what separates strong from average candidates.

Common Slow-Job Patterns

The Usual Suspects

▸Shuffle explosion: A join that produces a cartesian product due to duplicate keys
▸Data skew: One partition has 10B rows, others have 10K - one task takes 1000x longer
▸Small files: Reading 100K small Parquet files instead of 1K properly-sized files
▸Collect to driver: df.collect() or toPandas() pulling billions of rows to a single machine
▸UDF bottleneck: Python UDFs serialize/deserialize every row between JVM and Python

KEY TAKEAWAYS

Always start debugging from the Spark UI, not the code

Sort stages by duration and check shuffle metrics first

Task time skew (max >> median) indicates data skew

Shuffle spill to disk is always a red flag worth investigating

Name the debugging steps in order - interviewers grade your methodology

Optimize This Join

Daily Life

Interviews

Choose the right join strategy for a given data shape

"You're joining a 500GB fact table with a 2GB dimension table and it's slow. How do you fix it?" The answer they want: broadcast the dimension table. But the follow-ups go deeper.

Join Strategies in Spark

Strategy	When	Cost
Broadcast Hash Join	One side < 10MB (default)	No shuffle. Fastest.
Sort-Merge Join	Both sides large, keys sortable	Shuffle + sort both sides
Shuffle Hash Join	One side much smaller (but > broadcast)	Shuffle both, build hash on small side
Cartesian / BNL	No join keys (cross join)	O(n×m). Almost always a mistake.

The broadcast threshold is controlled by spark.sql.autoBroadcastJoinThreshold, default 10MB. If one side of the join is below this threshold, Spark ships the entire table to every executor, eliminating the shuffle entirely. For dimension tables up to ~1-2GB, you can force a broadcast even above the threshold.

	from pyspark.sql.functions import broadcast

	# Force broadcast join - eliminates shuffle entirely
	result = fact_df.join(
	broadcast(dim_df),
	on="dim_key",
	how="left"
	)

	# Verify the plan shows BroadcastHashJoin
	result.explain()

Bucketed Joins

When both sides are large and you join them repeatedly on the same key, bucketing eliminates the shuffle at read time. You pre-shuffle the data at write time so that matching keys land in the same bucket file. Subsequent joins on the bucket key skip the shuffle entirely.

	# Write both tables bucketed on the join key
	orders.write \
	.bucketBy(256, "customer_id") \
	.sortBy("customer_id") \
	.saveAsTable("orders_bucketed")

	customers.write \
	.bucketBy(256, "customer_id") \
	.sortBy("customer_id") \
	.saveAsTable("customers_bucketed")

	# Join with zero shuffle
	result = spark.table("orders_bucketed").join(
	spark.table("customers_bucketed"),
	on="customer_id"
	)

Bucketing has a cost: both sides must have the same number of buckets, and the data must be rewritten. Use it when the same join runs daily (e.g., in a pipeline), not for ad hoc queries.

Join Reordering

When joining three or more tables, order matters. Join the smallest result set first. If A (1B rows) joins B (10M rows) producing 1B rows, then joins C (500M rows) - that second shuffle moves 1B rows. But if you join A with C first and the predicate filters to 100M rows, the second shuffle moves 100M instead. Catalyst usually handles this, but check the plan when performance is bad.

TIP

In the interview, always ask about table sizes before proposing a join strategy. Saying 'How big is the other table?' shows you think in terms of data characteristics, not just syntax.

KEY TAKEAWAYS

Broadcast join eliminates shuffle - use it for any table under ~1-2GB

Default broadcast threshold is 10MB; override with spark.sql.autoBroadcastJoinThreshold

Bucketed joins pre-shuffle data at write time for zero-shuffle reads

Join order matters for multi-way joins - smallest intermediate result first

Always check the physical plan with .explain() to confirm the strategy Spark chose

How Do You Size the Cluster?

Daily Life

Interviews

Size a Spark cluster for a given workload

"You need to process 2TB of data daily. How do you size your Spark cluster?" This tests whether you understand memory, cores, and executors as interacting constraints rather than independent knobs.

The 5-Core Rule

Use 5 cores per executor. This is the well-tested sweet spot. More than 5 cores causes excessive GC pressure and HDFS throughput bottlenecks (each core opens concurrent connections). Fewer than 5 underutilizes memory. On a node with 16 cores, run 3 executors (5 cores each, 1 core reserved for OS/YARN).

Node Spec	Executors per Node	Cores per Executor	Memory per Executor
16 cores, 64GB	3	5	~19GB (after OS + overhead)
32 cores, 128GB	6	5	~19GB
8 cores, 32GB	1	5 (waste 2)	~28GB
4 cores, 16GB	1	4 (suboptimal)	~14GB

Memory Breakdown

Executor memory splits into three regions. Unified memory (default 60% of heap) handles both execution (shuffles, sorts, aggregations) and storage (cached DataFrames). Reserved memory (300MB) is off-limits. User memory (remaining 40%) holds your UDF objects and data structures. When execution needs more space, it can evict cached data. When both are full, you spill to disk.

Memory Config Cheat Sheet

▸spark.executor.memory = heap size (e.g., 19g)
▸spark.executor.memoryOverhead = off-heap + container padding (default max(384MB, 10% of heap))
▸spark.memory.fraction = 0.6 (unified memory share of heap)
▸spark.memory.storageFraction = 0.5 (initial storage share of unified memory, but boundary is soft)

Sizing for the Workload

For a 2TB daily job: the data is compressed on disk (typically 3-5x with Parquet + Snappy). In memory, expect 6-10TB. With 128MB target partitions, you need ~16,000 partitions. With 5 cores per executor and 20 executors, you have 100 task slots - so 160 waves of tasks. The job is bounded by shuffle I/O and executor count, not raw CPU.

KEY TAKEAWAYS

5 cores per executor is the standard starting point

Memory splits: 60% unified (execution + storage), 40% user, 300MB reserved

Scale executor count first, then size - horizontal beats vertical

Back-of-envelope: data_size_compressed × decompression_ratio / 128MB = partition count

Always state the tradeoff: more executors = more cost, fewer = longer runtime

What About Data Skew?

Daily Life

Interviews

Detect and resolve data skew in Spark joins

"Your join is fast for most keys but one key takes 10x longer. What is happening and how do you fix it?" Data skew is the single most common root cause of slow Spark jobs in production. Every interviewer expects you to handle it.

Detecting Skew

In the Spark UI, skew shows up as one task running dramatically longer than others in the same stage. The Tasks tab shows the min, median, and max task duration. If max is 50x the median, one partition holds massively more data than the others. Check the shuffle read size per task - the long task will show orders of magnitude more bytes.

	# Detect skew: find the heavy keys
	df.groupBy("join_key") \
	.count() \
	.orderBy(col("count").desc()) \
	.show(20)

	# If the top key has 10M rows and the median is 1K,
	# you have a skew problem

Salting: The Standard Fix

Salting splits a hot key into N artificial keys, distributing its rows across N partitions. Add a random salt (0 to N-1) to the skewed side, replicate the other side N times with each salt value, then join on the composite key. This turns one 10M-row partition into N partitions of ~10M/N rows each.

	from pyspark.sql.functions import lit, rand, floor, col, explode, array

	N_SALT = 10

	# Salt the large (skewed) side
	salted_large = large_df.withColumn(
	"salt", floor(rand() * N_SALT).cast("int")
	)

	# Replicate the small side with all salt values
	salted_small = small_df.withColumn(
	"salt", explode(array([lit(i) for i in range(N_SALT)]))
	)

	# Join on original key + salt
	result = salted_large.join(
	salted_small,
	on=["join_key", "salt"],
	how="inner"
	).drop("salt")

Other Skew Strategies

•Isolate Hot Keys

Filter out the skewed key(s)
Process them separately (broadcast the small side for just that key)
Union the results back together
Best when only 1-3 keys are skewed

•Repartition

repartition(N, 'key') before the join
Redistributes data more evenly
Adds a shuffle but can help with moderate skew
Not effective when one key dominates

Null keys are a common source of hidden skew. If 30% of your rows have a null join key, they all land in the same partition. Filter nulls before the join, process them separately, then union the results.

TIP

In the interview, mention null key skew unprompted. Most candidates forget it. Saying 'I'd also check for null keys - they're a common source of hidden skew' earns bonus points every time.

KEY TAKEAWAYS

Skew = one partition has orders of magnitude more data than others

Detect it: Spark UI task distribution, or groupBy + count + orderBy desc

Salting splits hot keys across N artificial partitions

Isolating hot keys works when only a few keys are problematic

Null join keys are the most common hidden skew - filter them first

❯❯❯PUTTING IT ALL TOGETHER

> The interviewer says: "this Spark job joins a 500GB fact table to a 2GB dimension table. It used to take 20 minutes and now it takes 3 hours. Debug it out loud, then tell me how you would size the cluster for it." This is the whole gauntlet in one prompt.

Start at the Spark UI Stages tab sorted by shuffle write rather than at the code, because 80% of slow jobs are shuffle-bound and one stage writing 500GB while others write 5GB localizes the problem in seconds.

Click into that stage and read the task duration distribution: uniform durations point at partition sizing or spill, while one task at 45 minutes against 199 at 30 seconds is skew by definition.

Confirm skew by grouping on the join key and ordering by count descending, then fix it with salting: add a random salt of 0 to N-1 on the large side, explode the small side across all N salt values, and join on the composite key. Filter null join keys out first since they all collapse into one partition.

The 2GB dimension is above the default spark.sql.autoBroadcastJoinThreshold of 10MB, so force broadcast(dim_df) to remove the shuffle entirely and verify BroadcastHashJoin in explain(), or bucket both sides on the join key if this same join runs daily.

Any shuffle spill in that stage means partitions are too large or executor memory is too low, so size partitions to roughly 128MB, which for 500GB is about 4,000 shuffle partitions rather than the default 200.

Close on cluster sizing with the interacting constraints: 5 cores per executor, executor memory split into the 60% unified region plus overhead, and enough executors that the wave count over your partition target is reasonable, since the job is bounded by shuffle I/O and executor count rather than raw CPU.

KEY TAKEAWAYS

Driver plans and schedules, executors run tasks on partitions - stages split at shuffle boundaries

Debug slow jobs from the Spark UI stages tab: sort by duration, check shuffle metrics, examine task distribution for skew

Broadcast joins eliminate shuffles for tables under ~1-2GB; bucketed joins eliminate shuffles for repeated joins on the same key

5 cores per executor, ~19GB memory, scale executor count before executor size

Data skew is the #1 root cause of slow Spark jobs - detect with task distribution, fix with salting or hot key isolation

The technical gauntlet every pipeline interview hits

Category: Pipeline Architecture
Difficulty: intermediate
Duration: 35 minutes
Challenges: 0 hands-on challenges

Topics covered: Explain Spark Architecture, This Job Is Slow, Optimize This Join, How Do You Size the Cluster?, What About Data Skew?

Lesson Sections

Explain Spark Architecture (concepts: paSparkExecutionModel)
"Walk me through how Spark executes a query." This is the opener. If your answer is vague, the interviewer downgrades your level. If it is precise, they trust your debugging answers later. The Driver-Executor Model Spark runs one driver process and N executor processes. The driver parses your code into a logical plan, optimizes it via Catalyst, converts it to a physical plan, and splits that plan into stages. Each stage contains tasks - one task per partition. The driver sends tasks to executo
This Job Is Slow (concepts: paSparkUiDiagnosis)
"This Spark job used to take 20 minutes. Now it takes 3 hours. What do you do?" This is the most common Spark question across all companies. The interviewer is testing your debugging methodology, not a single trick. The Debugging Sequence Check shuffle metrics FIRST. 80% of slow Spark jobs are shuffle-bound. Open the Spark UI → Stages tab → sort by shuffle write. If one stage writes 500GB of shuffle data while others write 5GB, you found the problem. Do not start with code review - start with
Optimize This Join (concepts: paBroadcastJoin)
"You're joining a 500GB fact table with a 2GB dimension table and it's slow. How do you fix it?" The answer they want: broadcast the dimension table. But the follow-ups go deeper. Join Strategies in Spark The broadcast threshold is controlled by spark.sql.autoBroadcastJoinThreshold, default 10MB. If one side of the join is below this threshold, Spark ships the entire table to every executor, eliminating the shuffle entirely. For dimension tables up to ~1-2GB, you can force a broadcast even above
How Do You Size the Cluster? (concepts: paSparkExecutionModel)
"You need to process 2TB of data daily. How do you size your Spark cluster?" This tests whether you understand memory, cores, and executors as interacting constraints rather than independent knobs. The 5-Core Rule Use 5 cores per executor. This is the well-tested sweet spot. More than 5 cores causes excessive GC pressure and HDFS throughput bottlenecks (each core opens concurrent connections). Fewer than 5 underutilizes memory. On a node with 16 cores, run 3 executors (5 cores each, 1 core reser
What About Data Skew? (concepts: paDataSkew)
"Your join is fast for most keys but one key takes 10x longer. What is happening and how do you fix it?" Data skew is the single most common root cause of slow Spark jobs in production. Every interviewer expects you to handle it. Detecting Skew In the Spark UI, skew shows up as one task running dramatically longer than others in the same stage. The Tasks tab shows the min, median, and max task duration. If max is 50x the median, one partition holds massively more data than the others. Check the