PySpark Drop Duplicates and Window Dedup (2026)
dropDuplicates triggers a full shuffle. distinct hashes all N columns. Window dedup is the only method that gives you deterministic control over which row survives. Interviewers focus on whether you know the tradeoffs between all three.
dropDuplicates() on Specific Columns
# Interviewers test: do you know which row survives?
# Answer: it is NOT deterministic. The surviving row is arbitrary.
deduped = df.dropDuplicates(["customer_id", "order_date"])
# dropDuplicates hashes only the columns you specify.
# This triggers a full shuffle (wide transformation).
# Shuffle cost scales with the number of columns you carry,
# not the number of columns in the dedup key.dropDuplicates triggers a full shuffle. Spark redistributes every row by the specified columns so identical keys land on the same partition. The row that survives is arbitrary. If your interviewer asks "which record do you keep?", the answer is: you do not control it without a window function.
distinct() for Full-Row Deduplication
# distinct() hashes ALL columns in the DataFrame.
# On a 40-column table, that means hashing 40 fields per row.
deduped = df.distinct()
# Equivalent to dropDuplicates with no arguments:
deduped = df.dropDuplicates()
# If you only care about 3 columns being unique,
# distinct() wastes work hashing the other 37.
# Use dropDuplicates(["col1", "col2", "col3"]) instead.distinct() hashes all N columns. On wide tables this is expensive. The tradeoff: distinct guarantees full-row uniqueness, while dropDuplicates(subset) only guarantees uniqueness on the subset columns. Interviewers focus on whether you understand that difference and pick the right one.
Window Function Deduplication (Deterministic Row Selection)
from pyspark.sql import functions as F
from pyspark.sql.window import Window
# Interviewers expect this pattern for "keep the latest record"
w = Window.partitionBy("customer_id").orderBy(F.desc("updated_at"))
deduped = (
df
.withColumn("rn", F.row_number().over(w))
.filter(F.col("rn") == 1)
.drop("rn")
)
# Why row_number and not rank?
# rank() assigns the same number to ties, so you might keep 2+ rows.
# row_number() always picks exactly one row per partition.This is the only dedup method that gives you deterministic control over which row survives. You choose the orderBy column. Interviewers look for: partitionBy on the dedup key, orderBy on the tiebreaker, row_number (not rank), and filter to rn == 1. If you use rank instead of row_number, you can still get duplicates when ties exist.
Drop Duplicate Columns After a Join
# After a join, both tables contribute their join key column.
# This creates ambiguous references.
result = orders.join(customers, orders.id == customers.id)
# result now has two "id" columns
# Fix 1: Use list syntax. Spark keeps only one copy of the key.
result = orders.join(customers, on=["id"], how="inner")
# Fix 2: Drop the duplicate explicitly.
result = orders.join(customers, orders.id == customers.id) \
.drop(customers.id)
# Fix 3: Alias and select.
result = orders.alias("o").join(customers.alias("c"), "id") \
.select("o.*", F.col("c.name").alias("customer_name"))Duplicate column names after joins break downstream operations like .select("id"). The list syntax on=["id"] is the cleanest fix because Spark automatically deduplicates the join key. A strong interview answer mentions all 3 approaches and explains when each is appropriate.
Know PySpark deduplication the way the interviewer who asks it knows it.
Deduplication is where candidates discover that dropDuplicates and a window-ranked row_number are not interchangeable at scale. That distinction runs through the PySpark interview questions, and the PySpark practice problems shows which loops weight Spark at all.
Dedup questions rarely arrive on their own. They show up inside a larger prompt about a pipeline that double-counts after a replay, which means the follow-up is about idempotency and late-arriving data rather than the API call itself. The Spark mock interview runs that longer conversation, so you practice defending a dedup key instead of just naming one.
The Shortest Route
> A message-routing service keeps an adjacency dict mapping each node to the nodes it can forward to directly. Given that `graph` and two node names, return the fewest hops from `start` to `end`, or -1 when `end` can't be reached.
Sample input & expected output(2 examples)
PySpark Drop Duplicates FAQ
What is the difference between dropDuplicates and distinct in PySpark?+
Is dropDuplicates deterministic in PySpark?+
Does dropDuplicates trigger a shuffle?+
Practice PySpark Dedup Patterns Before Your Interview
- 01
Reading a solution is not the same as writing one
Every engineer who has frozen on a query they had read a dozen times knows the gap. The only preparation that closes it is producing the answer yourself, under time, before the interview does it for you
- 02
76% of hiring managers reject on the coding task, not the resume
From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice
- 03
5 problem shapes cover 80% of data engineer loops
Parsing and reshaping, sessionization, dedup with tie-breaks, streaming aggregation, top-N-per-group. Writing them by hand turns the unfamiliar into pattern recognition
Related PySpark Dedup Guides
Join types, broadcast joins, and duplicate column handling
Aggregations, shuffle boundaries, and skew handling
The questions data engineers face in real interviews
1 data engineering challenge a week on dirty production-shaped data, scored blind on a hidden dataset. Submit before the freeze; results at the reveal.