PySpark Interview Questions for Data Engineers (2026)

select() projects specific columns and can rename or transform them. withColumn() adds or replaces a single column while keeping all others. Use select() when you want to control exactly which columns appear in the output. Use withColumn() for adding derived columns. A strong answer notes that chaining many withColumn() calls is inefficient because each creates a new projection; select() with multiple expressions is better.

This is data skew. One partition holds disproportionately more data than others. Fixes include salting the join key (appending a random suffix, joining on the salted key, then aggregating), broadcasting the smaller table, or using AQE skew join optimization (spark.sql.adaptive.skewJoin.enabled). A strong answer mentions checking the partition sizes in the Spark UI to confirm the diagnosis before applying a fix.

repartition(n) performs a full shuffle to create exactly n partitions with roughly equal data. coalesce(n) merges existing partitions without a shuffle, so it can only reduce partition count. coalesce causes problems when reducing from many partitions to very few: some executors end up with much more data than others, creating skew. It also cannot increase partition count.

First, check if a built-in function can replace the UDF (regexp_extract, regexp_replace). If not, convert to a Pandas UDF (vectorized) to process Arrow batches instead of row-by-row serialization. If the regex is complex, consider precompiling the pattern outside the UDF. A strong answer quantifies the cost: Python UDFs force serialization between JVM and Python for every row, which can be 10x to 100x slower than native Spark functions.

Window functions operate on a partition of rows defined by a WindowSpec. You define partitionBy (grouping), orderBy (sorting), and optionally rowsBetween or rangeBetween. The computation runs without collapsing rows. A strong answer includes: from pyspark.sql.window import Window; w = Window.partitionBy('dept').orderBy(F.desc('salary')); df.withColumn('rank', F.rank().over(w)).

cache() marks the DataFrame for storage in executor memory (MEMORY_AND_DISK by default). The data is materialized on the first action and reused for subsequent actions. Avoid caching when the DataFrame is used only once, when it is very large relative to available memory, or when the upstream computation is cheap. A strong answer mentions unpersist() to free memory explicitly and notes that caching serialized data (MEMORY_ONLY_SER) reduces memory footprint at the cost of CPU.

Use watermarks. withWatermark('event_time', '10 minutes') tells Spark to drop data older than 10 minutes past the latest event time seen. This allows stateful aggregations to discard stale state. A strong answer discusses the tradeoff: a shorter watermark frees memory faster but drops more late data; a longer watermark retains more state but uses more memory. Mention that output mode (append vs update vs complete) affects when results are emitted.

Broadcast the 50MB dimension table. Use F.broadcast(dim_df) or rely on autoBroadcastJoinThreshold. This sends the small table to every executor and avoids shuffling the 500GB fact table entirely. A strong answer mentions verifying the dimension table size is accurate (not just the on-disk size, which may differ from in-memory size after deserialization), and notes that if the dimension table grows past the threshold, the job will silently switch to a sort-merge join with a full shuffle.

Catalyst is Spark SQL's query optimizer. It converts logical plans to optimized physical plans through rule-based and cost-based optimization. It handles predicate pushdown, column pruning, join reordering, and constant folding. PySpark DataFrame operations go through Catalyst; RDD operations do not. This is a key reason DataFrames outperform RDDs for structured data. A strong answer notes that Catalyst cannot optimize Python UDFs because it treats them as black boxes.

map() applies a function to each row individually. mapPartitions() applies a function to an entire partition iterator, which is more efficient for operations that have setup cost (database connections, model loading). foreach() is an action that applies a function for side effects without returning data. Choose mapPartitions() when you need to amortize expensive initialization across rows. Choose foreach() or foreachPartition() for writing to external systems.