Anti-Join: Intermediate
NOT EXISTS vs NOT IN vs LEFT JOIN IS NULL
Spot anti-join patterns: users who never purchased, products never viewed, events with no match.
- ▸"users who have never X"
- ▸"products that have never been Y"
- ▸"customers without a Z"
- ▸"events with no matching ..."
- ▸"records present in A but not in B"
- ▸Any question framed as the absence of a relationship
Why the choice between forms matters
All three forms answer the same English question. They are not interchangeable. NOT IN has a NULL trap that returns zero rows on the wrong input. LEFT JOIN WHERE NULL is verbose but predictable. NOT EXISTS is the cleanest semantically but unfamiliar to candidates who learned SQL from join-heavy tutorials. The interviewer is not asking which one you like. They are asking which one you would write at three in the morning when the dashboard is on fire and the source table just learned how to produce NULLs.
- ▸NOT IN: ships zero rows when the subquery has a NULL
- ▸LEFT JOIN WHERE NULL: verbose, requires picking which column to check IS NULL
- ▸NOT EXISTS: NULL-safe, reads like the English question, default reach
Default to NOT EXISTS. The reason is that it handles NULLs in the subquery correctly without an extra clause, reads as 'this thing does not exist,' which matches the English question, and produces the same query plan as NOT IN on most modern engines without the NULL hazard. Saying 'I default to NOT EXISTS because it is NULL-safe and reads like the question' is the answer the interviewer is checking for.
What the interviewer is actually testing
- Writes NOT IN without checking whether the subquery can return NULL
- Dashboard returns the right rows in dev (no NULLs) and zero rows in prod (one NULL)
- Cannot explain the difference when on-call asks
- Gets the rollback PR after the postmortem
- Defaults to NOT EXISTS because the NULL handling is correct by construction
- Explains the NOT IN trap unprompted
- Defends the choice with the three-valued-logic argument
- Has the LEFT JOIN WHERE NULL form ready as a fallback when the engine plans NOT EXISTS poorly
The NOT IN NULL Trap That Returns Zero Rows
Write all three anti-join forms and know which the interviewer expects based on dialect and clarity.
- ▸Naming all three forms unprompted: floor
- ▸Defaulting to NOT EXISTS with a correctness reason: pass
- ▸Anticipating the NOT IN NULL trap before being asked: strong-hire signal
- ▸Naming the plan equivalence on modern engines: depth signal
Form 1: LEFT JOIN WHERE NULL
Form 2: NOT EXISTS
Form 3: NOT IN, and how to choose between the three
Choosing the Form the Interviewer Expects
Explain why NOT IN returns zero rows when the subquery contains a NULL, and how NOT EXISTS avoids this.
Why NOT IN with a NULL returns zero
SQL has three values: TRUE, FALSE, and UNKNOWN. Comparing anything to NULL returns UNKNOWN, not FALSE. NOT IN expands to a chain of inequalities: 'x NOT IN (1, 2, NULL)' becomes 'x != 1 AND x != 2 AND x != NULL.' That last comparison is UNKNOWN. UNKNOWN AND TRUE is UNKNOWN. UNKNOWN is not TRUE, so the row is excluded from the result. Every row is excluded because the chain always contains an UNKNOWN.
- ▸x NOT IN (1, 2, NULL) expands to x != 1 AND x != 2 AND x != NULL
- ▸x != NULL is UNKNOWN, not TRUE or FALSE
- ▸TRUE AND TRUE AND UNKNOWN is UNKNOWN
- ▸UNKNOWN is not TRUE, so the row is excluded
- ▸Every row is excluded by the same logic. Result set is empty.
The bug, demonstrated
Why NOT EXISTS avoids the trap
- Single NULL in the subquery returns zero rows total
- The bug is silent: no error, no warning, just an empty result
- Reproduces only on the data that has the NULL
- Standard interview trap; standard production incident
- NULL in the subquery is treated row-by-row, not as a poison value
- Correctly excludes only the customers who have a matching order
- Predictable on every data shape
- The reason it should be your default
The reason this trap exists in the language is that SQL was designed when 'unknown' was treated as a third truth value with explicit semantics. Every other reasonable interpretation would either silently corrupt data (treating NULL as a real value) or break backwards compatibility. Knowing this history is not required, but mentioning that NOT IN's behavior is a consequence of three-valued logic, not a bug, signals depth.
The fix when you cannot avoid NOT IN
| Situation | Phrasing that flatlines | Phrasing that lands |
|---|---|---|
| You see 'customers who never X' | "I'll use NOT IN." | "This is an anti-join. I default to NOT EXISTS because it handles NULLs in the subquery correctly and reads like the English question; NOT IN with a single NULL in the inner result returns zero rows." |
| The interviewer asks 'why not NOT IN' | "It's slower." | "Three-valued logic. x NOT IN (..., NULL) expands to a chain of inequalities; the comparison to NULL is UNKNOWN, UNKNOWN AND TRUE is UNKNOWN, every row is excluded. The bug is silent: empty result, no error." |
| The interviewer asks 'when LEFT JOIN WHERE NULL' | "When I feel like it." | "When the team's house style prefers it, or when I need columns from the right side that EXISTS can't expose. Otherwise NOT EXISTS reads closer to the question and avoids the IS-NULL-on-which-column question." |
| The interviewer asks 'how does it scale' | "It's fast." | "All three forms compile to the same hash anti-join on modern optimizers. The cost lever is whether orders.customer_id is indexed; without an index, the plan degrades to a scan. For a dashboard, I'd materialize the anti-join result on the orders pipeline cadence rather than recompute every refresh." |
| The interviewer adds a temporal window | "I'll add BETWEEN." | "BETWEEN prevents a pure hash anti-join; the optimizer falls back to nested-loop or merge. On a billion-row source I'd pre-aggregate page-B visits into a daily-window table first so the anti-join becomes an equi-join again." |
Multi-Column Anti-Joins and Composite Keys
Handle composite key anti-joins (e.g., users who visited page A but not page B on the same day).
The multi-column version
- All three forms work. NOT EXISTS is the safe default; LEFT JOIN WHERE NULL is the verbose fallback.
- NOT EXISTS scales cleanly. NOT IN multi-column requires tuple syntax and still hits the NULL trap on every column.
- NOT EXISTS is the only clean option. The BETWEEN clause turns the anti-join into a non-equi join, killing the hash plan.
The LEFT JOIN form scales awkwardly here
The NOT IN form does not generalize
The variant interviewers like: anti-join with a temporal window
When the "Missing" Set Is Itself Filtered
Discuss query plan differences between the three forms and when the optimizer treats them identically.
The three execution plans you should know
- Fastest plan for equi-joins with no range conditions
- Memory cost is the size of the right side's hash table
- Modern engines default to this when both sides fit in memory
- Produced by all three anti-join forms when the optimizer is well-tuned
- Fallback when the right side is small or no usable index exists
- Catastrophic on large tables because cost is multiplicative
- Produced by older engines that do not rewrite correlated subqueries
- Sign you need to rewrite the query or add an index
When the optimizer rewrites NOT IN, LEFT JOIN, and NOT EXISTS to the same plan
What to say when the plan is bad
- ▸Confirm the plan with EXPLAIN before any optimization
- ▸Ensure the right-side join column has an index
- ▸Precompute the anti-join result if it powers a dashboard with high refresh cadence
- ▸For multi-column or range-based anti-joins, NOT EXISTS is the only clean reach
The scale conversation, condensed
Mention precomputation unprompted when the interviewer asks about scale at the dashboard level. 'For a dashboard that refreshes hourly, I would materialize the customers-without-orders set as its own table, refresh it on the orders pipeline cadence, and serve the dashboard from the materialized table.' This sentence is the architectural beat that earns the credit at this level.
The closing summary
> You are in an Apple data engineering interview. The interviewer asks: 'Find customers who have never placed an order. The orders table sometimes has NULL customer_id from a bad ingestion run.'
NOT EXISTS for anti-joins: it is NULL-safe without an extra clause, it reads like the English question, and modern optimizers compile it to the same anti-join operator as the other two forms.NOT IN returns zero rows, not wrong rows, when the subquery contains a single NULL. Comparing 5 to NULL yields UNKNOWN, the conjunction collapses to UNKNOWN, and every candidate row is excluded. This is three-valued logic working as designed, not an engine bug.NOT IN, the fix is SELECT customer_id FROM orders WHERE customer_id IS NOT NULL inside the subquery. That correctness now depends on every future author remembering the filter, which is why NOT EXISTS is the safer default.NOT EXISTS is the only form that scales cleanly. Join the subquery on every column that defines the match, including b.visit_date = a.visit_date to enforce same-day semantics, and skip the question of which right-side column to test for IS NULL.b.visit_date BETWEEN a.visit_date AND a.visit_date + INTERVAL '7 days' makes the anti-join non-equi, so the engine cannot use a hash anti-join and falls back to nested loop or merge. Pre-aggregate the right side into a daily-window table when the source is large.EXPLAIN before optimizing, index the join column on the right side to avoid the O(N * M) nested loop, and materialize the anti-join result into its own table when it powers a dashboard that refreshes on a fixed cadence.Finding what is NOT there is harder than finding what is
- Category
- SQL
- Difficulty
- intermediate
- Duration
- 25 minutes
- Challenges
- 0 hands-on challenges
Topics covered: NOT EXISTS vs NOT IN vs LEFT JOIN IS NULL, The NOT IN NULL Trap That Returns Zero Rows, Choosing the Form the Interviewer Expects, Multi-Column Anti-Joins and Composite Keys, When the "Missing" Set Is Itself Filtered
Lesson Sections
- NOT EXISTS vs NOT IN vs LEFT JOIN IS NULL (concepts: sqlNullInJoins)
Here is the shape that recurs in every reporting interview. The interviewer has a customers table and an orders table. They say: 'find customers who have never placed an order.' The query is one of three forms, each four to six lines. The interview signal is not whether you can write the query. It is whether you pick the form that holds up under the conditions of the real data. Why the choice between forms matters Default to NOT EXISTS. The reason is that it handles NULLs in the subquery correct
- The NOT IN NULL Trap That Returns Zero Rows (concepts: sqlNullInJoins)
Three forms produce the same answer on clean data. Knowing all three, and being able to deliver each one from memory, is the floor. Knowing which one to reach for is what the interviewer is scoring. Form 1: LEFT JOIN WHERE NULL Join the two tables with a LEFT JOIN, then filter for rows where the right side is NULL. The NULL appears for rows in the left table that did not match any row in the right table. This is the form most candidates default to because it is the most explicit: you can see the
- Choosing the Form the Interviewer Expects (concepts: sqlThreeValuedLogic)
This section is the reason this question lives in interview loops. NOT IN with a NULL in the subquery returns zero rows. Not 'some' rows. Not 'wrong' rows. Zero. The dashboard goes to zero. The on-call gets paged. The fix is one character: switch NOT IN to NOT EXISTS. The trap exists because of how SQL's three-valued logic interprets NOT IN. Why NOT IN with a NULL returns zero The bug, demonstrated Walk through a customer who has never ordered. Their customer_id is 5. The subquery returns (1, 2,
- Multi-Column Anti-Joins and Composite Keys (concepts: sqlExists)
The escalation: 'find users who visited page A but never page B on the same day.' One condition is no longer enough; the anti-match has to consider multiple columns at once. The form choice matters again, and the same NULL trap reappears in a slightly different shape. The multi-column version Express the absence with NOT EXISTS, joining the subquery on every column that defines the match. The subquery joins on three columns: user_id, visit_date, and page. The visit_date join is what enforces 'on
- When the "Missing" Set Is Itself Filtered (concepts: sqlExists)
After correctness, the interviewer escalates to performance. The follow-up: 'how does the query planner execute this on a billion-row table?' All three forms can produce the same physical plan on a modern optimizer; they can also fall back to nested loops on an older one. Knowing when each plan applies, and what to do when the optimizer picks the slow one, is the depth this question rewards at the mid-and-above level. The three execution plans you should know Hash anti-join: build a hash table o