PySpark Coding Practice

Junior roles (L3 and L4) test whether you can write correct transformations. Most answers chain filter and select with withColumn and need an inner or left join. Aggregation means groupBy with sum (or count and avg). NULLs get handled with coalesce or fillna, sometimes isNull, and you will cast and rename columns along the way. The typical prompt is revenue per customer over a date range with some rows excluded, and the typical miss is a left join that silently introduces NULLs.

Mid level roles (L4 and L5) test whether you can handle messy data. Window functions come up constantly (row_number, rank, lag, running totals), and so does deduplication. Some prompts join 3 or more DataFrames. Others need a pivot or dates that cross time zones, and you should know when a UDF is the wrong tool. Expect a top N per group prompt with ties in it.

Senior roles (L5 and L6) test whether you can diagnose a slow job. You should know when a broadcast beats a sort merge join and how salting fixes data skew. Interviewers also ask about shuffle partition counts and spill, and about when to cache or checkpoint. Expect to read explain() output too. The prompt is often a stuck job described by its task metrics rather than a question about syntax.

Staff roles (L7 and up) meet Spark inside system design. The questions cover partition layout and file sizes and incremental processing with watermarks and merges. Executor sizing and cost come into it as well. The code matters less than the questions you ask about the data before you write it.

The standard junior prompt: given an orders DataFrame, compute each customer's total revenue for the last 90 days and exclude cancelled orders. It looks like 3 lines of code, and it hides 2 traps that interviewers check on purpose.

NULL comparison is the first trap. F.col("status") != "cancelled" evaluates to NULL for a row whose status is NULL, and filter drops NULL, so orders with a missing status vanish from the revenue. If those orders should count, write ~F.col("status").eqNullSafe("cancelled"), which treats NULL as a value, or ask the interviewer which behaviour they want.

The other trap is where the filter sits. Filter before you join, not after: a join moves both inputs across the network, and every row removed first is a row that is never shuffled. Catalyst pushes many simple filters below a join by itself, but a filter on a column computed after the join cannot move, and saying why shows you know what the optimizer does. After a left join, every column from the right side is NULL for unmatched rows, so a later sum or comparison on those columns needs a coalesce.

The most reused mid level prompt: for each product category, return the top 3 customers by spend last quarter, with each one's share of the category total. It tests 3 things at once: aggregating before ranking, choosing the ranking function, and computing the share before the filter removes the rows it depends on.

Choose the ranking function by what should happen on a tie. row_number gives every row a distinct number, so exactly 3 rows come back; add a tiebreaker column to the orderBy or the winner of a tie changes between runs. rank gives tied rows the same number and skips the next ones (1, 1, 3), so a 3 way tie for first returns 3 rows and a tie for third returns 4. dense_rank does not skip (1, 1, 2). Using rank where the prompt says exactly 3 is the most common miss.

Running totals carry a quieter trap. The PySpark Window documentation states that when a window has an orderBy, the default frame is a range frame from unbounded preceding to the current row. Rows that share an orderBy value are peers in a range frame, so they see each other. Take amounts of 10 on day 1 and 20 and 30 on day 2: the default frame returns 10, 60, 60, while rowsBetween(Window.unboundedPreceding, Window.currentRow) returns 10, 30, 60. Name the frame explicitly whenever the order column can repeat.

Deduplication questions come in 2 forms, and the right tool depends on which one you are given. When any copy of a duplicate row is acceptable, dropDuplicates(["order_id"]) gives a correct answer cheaply. It keeps 1 row per key, but it does not promise which one, and the row it keeps can change between runs.

If the prompt says keep the latest version of each record, which is the usual case for change data and event streams, dropDuplicates is wrong even when it happens to pass on sample data. Rank inside each key with row_number().over(Window.partitionBy("order_id").orderBy(F.col("updated_at").desc())) and keep row 1. Decide what happens when 2 versions share the same updated_at, and add a second ordering column so the answer is deterministic.

For a streaming DataFrame, dropDuplicatesWithinWatermark bounds the state Spark keeps for deduplication by the event time watermark. Mentioning that state is what grows without a watermark is the senior version of the same answer.

Spark joins 2 large DataFrames with a sort merge join: both sides are shuffled by the join key so matching keys land in the same partition, then each partition is sorted and merged. When 1 side is small, a broadcast hash join sends a full copy of it to every executor and the large side never moves. Spark chooses a broadcast by itself when a side's estimated size is under spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB.

That default is small on purpose, and interviewers test both directions. Broadcasting a 50 GB table is the classic wrong answer: every executor would need memory for all of it. The less obvious miss is a lookup that is too big for the default but easily small enough to broadcast. A 2M row lookup of 180 MB is 18 times the threshold, so Spark shuffles 800M rows to join it unless you write large.join(F.broadcast(lookup), "key") or raise the threshold.

Spark SQL accepts the same choice as a hint. When both sides of a join carry conflicting hints, Spark prefers /*+ BROADCAST(lookup) */ first, then MERGE, then SHUFFLE_HASH, and SHUFFLE_REPLICATE_NL last. A strong answer names the size of each side before it names a strategy, and the PySpark interview questions push on exactly that.

Senior interviewers often paste a physical plan and ask what it says. df.explain() prints it; df.explain(mode="formatted") splits it into a short tree and a numbered list of operator details, which is easier to talk through.

Read the plan from the bottom, where the scans are, to the top. On a scan, PushedFilters lists the predicates handed to the Parquet reader and PartitionFilters the ones that skip whole directories; an empty list where you expected a filter means the data is read in full. Every Exchange hashpartitioning(key, 200) is a shuffle by that key into 200 partitions. BroadcastExchange feeding a BroadcastHashJoin is a broadcast join; SortMergeJoin with a Sort and an Exchange under each side is the shuffle join.

At the top, AdaptiveSparkPlan isFinalPlan=false means you're looking at the plan before execution; AQE can still change the join strategy and the partition counts once runtime sizes are known. After an action, look again for the plan that actually ran, or find it in the SQL tab of the Spark UI. Saying that unprompted is the kind of detail the Spark optimization interview questions are built around.

A Python UDF is a black box to the Catalyst optimizer. Catalyst cannot push a filter through it or reorder operations around it, and it generates no code for it. Every row also has to travel from the JVM to a Python worker and back. Interviewers watch for a UDF written to do something pyspark.sql.functions already does. regexp_extract and when are common examples, along with date_trunc and split, or transform on an array.

When Python really is needed, prefer a pandas UDF, which moves data in Arrow batches and runs vectorised pandas code on each batch. Apache Arrow in PySpark notes that since Spark 4.2, Arrow optimization is also enabled by default for regular Python UDFs, which narrows the gap. It doesn't make a UDF visible to the optimizer, so the built-in function is still the first answer.

Keep the built-ins in reach while you practise. A 1 page PySpark functions cheat sheet saves more time than memorising rare ones, as long as it holds the date and string functions you reach for most, plus the conditional and array ones.

At staff level, the Spark question is a design question. A typical prompt: 2 TB of clickstream a day, a dashboard team that wants data within a minute, and a machine learning team that trains weekly on the full history.

Start by asking how late events arrive and how often records are corrected, because that decides between append and merge. Choose a partition layout that serves both readers, usually by event date, and target files in the hundreds of megabytes rather than thousands of small ones. Split the paths: a streaming job with a watermark for the dashboard, and a batch job for training that can be rerun for any date without double counting.

The miss at this level is optimising a query while the table layout underneath it is wrong, or proposing streaming without saying what happens to late data and duplicate deliveries.

A passing answer returns the right rows on the sample data. A strong answer starts by stating its assumptions: the grain of each input, whether keys can be NULL or duplicated, and what a tie should do.

It prices the solution: where the shuffles are, which side of each join is small, and what happens when 1 key holds a large share of the rows. It names the edge cases it handled, such as NULL keys after a left join or ties in a ranking, and says what late data does to the result. And it says what it would check on real data before trusting the result, such as a row count before and after each join to catch fan out.

None of that needs extra code. It needs practice saying it while typing, which is why a Spark mock interview with follow up questions trains something a solved problem alone does not.

Most PySpark question lists online are theory: RDD vs DataFrame, lazy evaluation, what a DAG is. They help with the quiz part of a screen and not with the coding part. For coding, you need problems with data, a way to run your answer, and a check that says whether it's right.

The site's catalogue holds 943 problems you can solve in PySpark or Spark SQL. Each is graded on submit and tagged with the employers that ask it. StrataScratch lets you answer many of its coding questions in PySpark, and the same questions take pandas or Polars as well as SQL. Its catalogue leans toward data science analysis. A local setup costs nothing: pip install pyspark runs Spark in local mode with Java 17 or later and Python 3.10 or later, and it is the only option that shows you real plans and real task metrics on your own data.

Whichever you use, solve each problem in the DataFrame API and then again in Spark SQL once. Interviewers accept either, and switching between them under time pressure is easier when you've done it before.