Spark Mock Interview for Data Engineers

Five years ago, you could land a senior data engineering role knowing SQL and Airflow. That bar has moved. The explosion of data volume (most mid-size companies now process 1-10TB daily) means every team needs at least one engineer who can debug a Spark job at 3 AM when the pipeline is stuck.

Spark interview questions test something different from SQL. SQL checks whether you can write correct queries. Spark checks whether you understand how data moves across a cluster. Can you read an explain plan and spot a skewed partition? Can you estimate how many executors you need for a 500GB dataset with 128MB partitions? Do you know the difference between a sort-merge join and a broadcast hash join, and when to force one over the other?

At Meta, Spark questions appear in 80% of L5+ data engineering loops. At Netflix, Spark is the primary processing framework, and every candidate is expected to reason about Spark execution during the system design round. Uber's data platform team gives a dedicated Spark coding round where you write PySpark in a shared editor. Google uses Spark less internally (they have Flume and Dataflow), but acquired Spark expertise through Dataproc and now tests it for cloud-facing roles.

The pattern is clear: if the role involves processing more than 100GB, Spark proficiency is a requirement, not a bonus.

The RDD vs DataFrame vs Dataset question shows up in nearly every Spark interview, usually within the first 10 minutes. Interviewers do not want a history lesson about Spark 1.x. They want to know if you understand the performance implications.

RDDs (Resilient Distributed Datasets) are Spark's original abstraction. They give you full control over data distribution and processing, but they bypass the Catalyst optimizer entirely. When you write an RDD transformation, Spark executes exactly what you wrote, with no query planning, no predicate pushdown, and no code generation. On a 1TB dataset, an RDD-based pipeline can be 10x slower than the equivalent DataFrame code because every operation involves serializing and deserializing Java objects.

DataFrames changed this. By expressing operations as a logical plan (similar to SQL), DataFrames let the Catalyst optimizer reorder operations, push filters down to the storage layer, and generate optimized Java bytecode through Project Tungsten. A DataFrame groupBy().agg() call does not execute immediately. Spark builds an execution plan, optimizes it, and then runs the physical operations. This is why the same logic written with DataFrames typically runs 3-10x faster than RDDs.

Datasets add compile-time type safety on top of DataFrames, but they only exist in Scala and Java. In PySpark, DataFrames are your only option (and the right one). The interviewer follow-up is usually: 'When would you still use RDDs?' The answer: custom partitioners, low-level control over data distribution, or when working with unstructured data that does not fit a schema.

DataDriven's Spark questions let you write both RDD and DataFrame versions of the same transformation, run them, and compare execution plans. You do not just memorize the answer. You see the difference.

Shuffles are the most expensive operation in Spark. A shuffle moves data across the network between executors, writes intermediate results to disk, and can easily double or triple your job's runtime. Every senior-level Spark interview includes at least one question about reducing or eliminating shuffles.

A common setup: you are shown a PySpark job that joins a 500GB fact table with a 2GB dimension table. The job runs for 3 hours. Why? Because Spark defaults to a sort-merge join, which shuffles both tables by the join key. The fix: broadcast the 2GB dimension table. With a broadcast join, Spark sends the small table to every executor, eliminating the shuffle entirely. Runtime drops to 20 minutes.

But interviewers go deeper. What if the 'small' table is 10GB? Spark's default broadcast threshold is 10MB (spark.sql.autoBroadcastJoinThreshold). You can increase it, but broadcasting 10GB to 200 executors means 2TB of network transfer. The right answer depends on cluster memory, network bandwidth, and whether the join is a one-time operation or runs hourly.

Data skew makes shuffles worse. If 30% of your data has the same join key (think: null values or a dominant user_id), one executor handles 30% of the work while others sit idle. The fix: salting. You add a random suffix to the skewed key, perform the join on the salted key, then aggregate the results. DataDriven's questions walk you through salting step by step, with real data that has intentional skew.

Other shuffle-reduction techniques that come up in interviews: using coalesce() instead of repartition() when reducing partition count (coalesce avoids a full shuffle), pre-partitioning data by the join key using bucketBy(), and restructuring multi-stage pipelines so expensive joins happen after aggressive filtering.

Spark caching is one of those topics that sounds simple but gets complicated fast. Calling .cache() on a DataFrame stores it in executor memory after the first action. Every subsequent action reuses the cached data instead of recomputing from source. Sounds great. The problem: executor memory is finite.

A 200GB DataFrame cached with MEMORY_ONLY across 50 executors with 4GB each means each executor needs to hold about 4GB of cached data. That leaves almost nothing for shuffles, aggregations, and other operations. The job runs slower, not faster. Interviewers test this by asking: 'You cached this DataFrame and the job got slower. Why?'

The answer involves understanding Spark's memory model. Executor memory is split between storage (for cached data) and execution (for shuffles and sorts). These pools share a unified memory region, and execution can evict cached blocks when it needs space. But if your cache is too large, the constant eviction and re-caching creates more overhead than it saves.

Interviewers also ask about persistence levels. MEMORY_ONLY is the default for .cache(). MEMORY_AND_DISK spills to local disk when memory is full, which prevents recomputation but adds I/O cost. MEMORY_ONLY_SER serializes the data, using less memory but requiring CPU for deserialization. Each level has a use case, and the right choice depends on data size, reuse frequency, and cluster configuration.

DataDriven's questions on caching are not theoretical. You write a pipeline with multiple stages, cache at different points, and observe how runtime changes. The AI evaluator checks whether your caching strategy actually improves performance for the given data size and cluster configuration.

Spark SQL lets you write SQL queries against DataFrames and Hive tables. The syntax is familiar. The execution model is not. A query that runs in 2 seconds on PostgreSQL might take 10 minutes on Spark because distributed execution adds coordination overhead. Conversely, a query that times out on PostgreSQL with 1B rows might complete in 30 seconds on a Spark cluster because the work is parallelized across 200 cores.

Interviewers test Spark SQL in two ways. The first: write a SQL query against a large dataset and explain how Spark will execute it. You need to discuss the logical plan (parsed SQL), the optimized logical plan (after Catalyst), and the physical plan (the actual operations). The second: debug a slow Spark SQL query by reading the explain() output and identifying bottlenecks.

Key Spark SQL concepts that come up repeatedly: Adaptive Query Execution (AQE), which re-optimizes the query plan at runtime based on actual data statistics. AQE can dynamically coalesce small partitions, switch join strategies, and handle data skew without manual intervention. It was introduced in Spark 3.0 and is on by default in Spark 3.2+. If you mention AQE in an interview without being prompted, it signals current, production-level experience.

DataDriven's Spark SQL questions give you a query, a schema, and data statistics. You predict the execution plan, run the query, compare your prediction to the actual plan, and then optimize. This prediction-first approach builds the kind of intuition that shows in interviews.

Most interview prep platforms cannot run Spark. They either syntax-check your PySpark code or tell you to set up a local environment. DataDriven is different. Every Spark question runs your code with a real PySpark session. You write code, execute it, see the output, and get AI feedback on both correctness and performance.

The execution environment includes a SparkSession with configurable settings. You can change partition counts, broadcast thresholds, and memory allocation. Want to see what happens when you set spark.sql.shuffle.partitions to 2 instead of 200? Run it and find out. Want to prove that your broadcast join is faster than a sort-merge join? The execution time is right there.

This matters because Spark intuition comes from running code, not reading documentation. You can read about data skew in a blog post and understand it conceptually. But until you have seen a Spark job hang at 99% progress because one partition has 10x more data than the others, you do not really understand it. DataDriven's questions create those situations intentionally.

The AI evaluator scores your Spark code across three dimensions: correctness (does it produce the right output?), efficiency (does it minimize shuffles and use appropriate join strategies?), and style (does it follow PySpark best practices like using built-in functions instead of UDFs?). You get line-by-line feedback explaining why a specific line creates a performance problem and what to do instead.

If you only study one Spark topic, make it broadcast joins. According to our data from 12,000+ mock interview sessions, broadcast join questions appear in 64% of Spark interview rounds. The concept is straightforward: instead of shuffling both tables to align on the join key, broadcast the smaller table to every executor. No shuffle. No network overhead for the large table. Dramatically faster.

The basic syntax is simple: df_large.join(broadcast(df_small), 'key'). But interviews go deeper. When is a broadcast join a bad idea? When the 'small' table is actually 5GB and you have 100 executors, you are sending 500GB across the network. When the small table is updated frequently and you are caching the broadcast variable, stale data becomes a correctness issue.

Interviewers also test edge cases. What happens when you broadcast a table with null join keys? (Those rows never match and are silently dropped in inner joins, which can cause data loss bugs.) What if the broadcast table is too large for driver memory? (The job crashes with an OutOfMemoryError during the broadcast phase, before any join processing begins.) What about broadcast joins with non-equi conditions? (Spark falls back to a nested loop join, which destroys performance.)

DataDriven has 15 dedicated broadcast join questions covering each of these scenarios. You start with the straightforward case, then progress to skewed data, multi-table joins, and situations where broadcasting is the wrong choice. Each question runs real PySpark, so you see the actual performance difference.