Filtering: Advanced
EXISTS takes a subquery and asks a single yes-or-no question about it: does this subquery return at least 1 row? The answer is TRUE as soon as a row turns up and FALSE when none do. It never reads the values in those rows, so putting SELECT 1 inside it gives the same result as SELECT * or SELECT order_id. Writing SELECT 1 is the convention because it tells the next reader that only existence matters.
| store_id | store_name |
|---|---|
| 1 | Maple Mugs |
| 2 | Fern Studio |
| 3 | North Knits |
| 4 | Pixel Prints |
| 5 | Tidal Tees |
| order_id | store_id | status |
|---|---|---|
| 101 | 1 | paid |
| 102 | 1 | refunded |
| 103 | 3 | paid |
| 104 | 5 | refunded |
| 105 | NULL | paid |
One existence check per store
SELECTs.store_id,s.store_nameFROM stores AS sWHERE EXISTS (SELECT1FROM orders AS oWHERE o.store_id = s.store_id)
Maple Mugs and North Knits survive, and so does Tidal Tees. Order 105 carries a NULL store_id, and o.store_id = s.store_id is never TRUE when one side is NULL, so that order matches no store and changes nothing here. Keep order 105 in mind: in the NOT IN section later in this lesson, the same row empties an entire result.
Narrow what counts as a match
Every condition you add to the subquery's WHERE narrows what counts as a related row. Adding AND o.status = 'paid' turns the question into: does this store have a paid order? Tidal Tees drops out, because its only order, 104, was refunded. Maple Mugs stays: order 102 was refunded, but order 101 is paid, and 1 paid row is enough.
The link can also combine several columns and comparisons. On a schema with launch dates, o.store_id = s.store_id AND o.created_at >= s.launched_at asks whether a store took an order after it launched. IN compares the outer column to the subquery's output by equality only, so a >= has to sit inside the subquery's own WHERE. EXISTS keeps both the equality and the >= in one WHERE that reads both tables.
- ▸Take 1 row from the outer table.
- ▸Fill its values into the columns the subquery borrows from it.
- ▸Run the subquery and stop at the first row it returns.
- ▸Keep the outer row if a row came back. Drop it if none did.
EXISTS next to IN
- The link lives inside the subquery's WHERE
- Can test several columns and conditions at once
- Reads no values, only whether a row came back
- Its NOT form stays correct when NULLs appear
- The subquery returns 1 column of values
- Compares a single outer column against that list
- Reads every value the subquery returns
- Its NOT form breaks when the list holds a NULL
A teammate deletes the store link from the subquery. Which stores does this query return?
NOT EXISTS flips the answer: it is TRUE when the subquery returns 0 rows for the current outer row. That makes it the standard tool for orphan hunting, finding rows on one side with no partner on the other. Stores with no orders, users who never logged in, invoices with no payment and products nobody reviewed all share this query shape.
Watch a missing partner keep a row
SELECTc.customer_id,c.companyFROM customers AS cWHERE NOT EXISTS (SELECT1FROM support_tickets AS tWHERE t.customer_id = c.customer_idAND t.status = 'open')
Where the condition sits changes the question
The status check belongs inside the subquery. There it redefines which partner rows count, so a store whose only order was refunded has no paid partner and stays. A condition in the outer WHERE filters the stores themselves, and stores has no status column to test. Each of the 3 tabs below asks a different question of the same 2 tables, which this section aliases as st and od.
> Support wants every customer who has never had a ticket closed. Using the customers and support_tickets tables from the animation, put the tiles in order. Two tiles are not needed.
NOT EXISTS next to EXCEPT
You already know a set-based way to find missing ids: SELECT store_id FROM stores EXCEPT SELECT store_id FROM orders returns 2 and 4. It answers the same question, with 3 differences that matter in practice.
- Returns whole outer rows, every column intact
- Keeps duplicate outer rows as they are
- The partner test can add any extra condition, such as a status or a date range
- Returns only the columns both queries select
- Removes duplicates from what it returns
- Compares entire rows for equality and nothing else
A typo compares the order to itself. Which stores does this return?
What NOT IN expands to
x NOT IN (a, b, c) is shorthand for x <> a AND x <> b AND x <> c. The subquery here returns 1, 1, 3 and 5, plus a NULL from order 105, so for store 2 the filter becomes 2 <> 1 AND 2 <> 1 AND 2 <> 3 AND 2 <> 5 AND 2 <> NULL. Any comparison with NULL yields NULL, which means unknown. An AND that holds a NULL and no FALSE is NULL as a whole. WHERE keeps a row only when its condition is TRUE, so store 2 is dropped. The table runs the same check for all 5 stores.
| store_id | vs 1, 1, 3, 5 | vs NULL | NOT IN result | kept? |
|---|---|---|---|---|
| 1 | FALSE (1 <> 1) | NULL | FALSE | no |
| 2 | all TRUE | NULL | NULL | no |
| 3 | FALSE (3 <> 3) | NULL | FALSE | no |
| 4 | all TRUE | NULL | NULL | no |
| 5 | FALSE (5 <> 5) | NULL | FALSE | no |
NOT IN against an allowlist with a NULL
SELECTuser_idFROM loginsWHERE user_id NOT IN (SELECTuser_idFROM risk_allowlist)
NOT EXISTS ignores the NULL order
For store 2, the NOT EXISTS subquery tests each order in turn: 1 = 2 is FALSE, 1 = 2 is FALSE, 3 = 2 is FALSE, 5 = 2 is FALSE and NULL = 2 is NULL. The inner WHERE also keeps only TRUE rows, so the subquery returns nothing and NOT EXISTS is TRUE. An unknown comparison can never make a subquery nonempty, so the NULL order has no effect. IN is safe for the opposite reason: it expands to comparisons joined by OR, and once a value matches, the OR is TRUE whatever else the list holds.
- ▸A subquery column that allows NULL, such as an optional foreign key
- ▸A literal list:
status NOT IN ('refunded', NULL)returns 0 rows on any table - ▸A NULL on the outer side:
NULL NOT IN (1, 2)is NULL, so that row drops even against a clean list - ▸A column that is clean today and gains NULLs after a schema change next month
- Write exclusions as NOT EXISTS with the link inside the subquery
- Use NOT IN only with a literal list you typed and can read
- Treat a 0-row exclusion result as a bug until you prove otherwise
- Expect NOT EXISTS to keep outer rows whose own key is NULL
- Run NOT IN against a subquery on a column that allows NULL
- Put NULL in a literal NOT IN list to mean no value
- Trust a NOT IN that passed testing: tomorrow's load may add a NULL
- Read a missing error as proof that the filter worked
Which stores does this query return?
LIKE knows 2 wildcards, % for any run of characters and _ for exactly 1, and its pattern must cover the whole string. That handles prefix and suffix checks. It cannot ask for exactly 4 digits, and it has no way to express a letter-or-digit choice or 1 or more of something. Regular expressions can, and REGEXP_LIKE(text, pattern) returns TRUE when the pattern matches the text.
In this lesson's SQL a backslash inside a string literal reaches the regex engine as typed, so '\d' arrives as \d and means a digit.
One difference catches people coming from LIKE: REGEXP_LIKE matches anywhere in the string. REGEXP_LIKE('SUMMER25', '25') is TRUE because 25 appears somewhere inside it. To demand that a pattern covers the whole value, anchor it with ^ for the start and $ for the end.
The building blocks
| Piece | What it matches | Example |
|---|---|---|
| BF | Those characters, in that order | BF matches inside BF-24-40 |
| . | Any 1 character | V.P matches VIP |
| [0-9] or \d | 1 digit | \d\d matches 25 in SUMMER25 |
| [A-Z] | 1 uppercase letter in the range A to Z | [A-Z] fails on summer25 |
| \s | 1 whitespace character | \s matches the space in VIP 10 |
| + | 1 or more of the piece before it | \d+ matches 2024 |
| * | 0 or more of the piece before it | BF-\d* matches BF- with or without digits |
| ? | 0 or 1 of the piece before it | VIP ?10 matches VIP 10 and VIP10 |
| {n} and {n,m} | Exactly n, or n to m, of the piece before it | \d{4} matches 2024 |
| ^ and $ | Start and end of the value | ^BF matches BF only at the start |
Merchants type discount codes into a free-text field, so the 6 codes below follow no single format. The animation tests each against ^[A-Z]+\d+$: uppercase letters followed by digits, at least 1 of each, with nothing else in the string.
SELECTcode_id,codeFROM discount_codesWHERE REGEXP_LIKE(code, '^[A-Z]+\d+$')
Only SUMMER25 passes. summer25 fails at its first character, because [A-Z] covers uppercase only. FREESHIP has letters but no digit, and \d+ needs at least 1. VIP 10 has a space between its letters and digits that no part of the pattern allows, and both BF codes stop at their dashes.
Anchors decide what a pass means
Leave out the $ and the rule weakens without changing any of these rows: ^BF-\d{4}-\d{2} also accepts BF-2024-405, because the first 2 digits after the second dash satisfy \d{2} and nothing checks what follows. An unanchored pattern only checks that a matching piece exists somewhere in the value.
- Must match the whole value
- 2 wildcards: % and _
- Can't count repetitions or name a set of characters
- Fine for fixed prefixes such as 'BF-%'
- Matches anywhere unless anchored
- Supports character classes and quantifiers, plus anchors and alternatives
- Counts characters exactly with {n}
- Built for format rules such as ^BF-\d{4}-\d{2}$
Which discount codes does this filter keep?
^(BF|CM)-\d{4}-\d{2}$| separates alternatives and the parentheses limit them, so the prefix may be BF or CM and nothing else.
^[^@\s]+@[^ ] negates a class: 1 or more characters that are neither @ nor whitespace, then an @.
(\.[a-z]+)+$A group followed by + repeats as a unit: 1 or more blocks of a dot and letters at the end.
^shop\.io$A backslash before a dot makes it literal. Unescaped, the dot would match any character.
(?i)^vip\d+$(?i) at the start makes the whole pattern ignore letter case.
From one pattern to a rule set
tidal tees@shop.io stops at the first WHEN. hello@fernstudio has no dot after its @, so the group (\.[a-z]+)+ has nothing to repeat. Pixel.Prints@gmail.com passes the domain rule because that rule carries (?i). The uppercase rule lacks it, so the address trips there. That label marks cleanup work rather than a delivery failure: the address still receives mail, and the row goes to a normalization step. Stores 1 and 3 fall through to ELSE.
- ▸One rule per failure reason, each a single REGEXP_LIKE
- ▸The most severe rule first, since CASE stops at the first TRUE
- ▸Anchors on every rule that describes a whole value
- ▸An explicit (?i) wherever case shouldn't matter, so the choice is visible
- ▸The rules saved as a view, a stored query that every downstream query selects from like a table
Capstone: who gets the Black Friday email
Stores 2 and 4 pass the NOT EXISTS. hello@fernstudio fails the pattern, so only Pixel Prints gets the email. Inside a character class, . is literal wherever it sits and needs no backslash, which is why [a-z0-9._] accepts the dot in Pixel.Prints. A - between 2 characters forms a range, as in a-z, so it is literal only as the first or last character of the class, which is why [a-z0-9-] puts it last.
SELECT REGEXP_LIKE('Pixel.Prints@gmail.com', '[A-Z]'). Remove pieces of the pattern one at a time until the result flips: the last piece removed is the one refusing the value.Remove (?i) from the capstone's pattern and change nothing else. What does the query return?
> A marketplace's lifecycle team builds a weekly win-back list: sellers with no order in the last 90 days, no open dispute, and a contact email that will deliver. The list feeds the email tool directly, so an empty list or a wrong one costs revenue before anyone notices.
Test for related rows, dodge the NULL inside NOT IN, and validate messy text with regular expressions.
Subqueries, regex, and other crimes
- Category
- SQL
- Difficulty
- advanced
- Duration
- 19 minutes
- Challenges
- 19 hands-on challenges
Topics covered: Correlated subqueries (EXISTS), NOT EXISTS for missing rows, NOT IN vs NULL gotchas, REGEXP_LIKE patterns, Regex operators and patterns
Lesson Sections
- Correlated subqueries (EXISTS) (concepts: Existence Checks)
The form you will reach for most is correlated: the subquery mentions a column from the outer query, so it's asked again for every outer row with that row's values filled in. For store 1 it reads as orders whose store_id is 1, and for store 2 as orders whose store_id is 2. Each outer row gets its own answer, and that answer decides whether the row stays. This section and the next 2 use the 5 stores and 5 orders below. One existence check per store Press play to watch the check run once per store
- NOT EXISTS for missing rows (concepts: Existence Checks)
Watch a missing partner keep a row The animation runs this shape on a support question: which customers have no open ticket? Acme and Initech each have an open ticket, so their subqueries return a row and NOT EXISTS is FALSE. Globex has only a closed ticket, which the status condition inside the subquery filters out, so its subquery comes back empty. Umbrella has no tickets at all. Globex and Umbrella stay. Where the condition sits changes the question The third tab shows why these filters compo
- NOT IN vs NULL gotchas (concepts: Three-Valued Logic)
Back to the Black Friday email. The natural first draft asks for stores whose id isn't IN the list of order store ids, and it returns 0 rows even though Fern Studio and Pixel Prints have never sold anything. The cause is order 105 and its NULL store_id. What NOT IN expands to Stores that have orders come out FALSE, which is the intended answer. Stores without orders come out NULL, which WHERE treats the same as FALSE. Every row loses, and the query raises no error. NOT IN against an allowlist wi
- REGEXP_LIKE patterns (concepts: Regex Matching)
In this lesson's SQL a backslash inside a string literal reaches the regex engine as typed, so '\d' arrives as \d and means a digit. The building blocks Anchors decide what a pass means Black Friday codes should look like BF-2024-40: the letters BF and a 4-digit year, then a 2-digit discount, with a dash between each part. The query below runs 2 rules side by side. LIKE 'BF-%' checks only the prefix, so it lets the 2-digit year in BF-24-40 through. The anchored pattern checks every character fro
- Regex operators and patterns (concepts: Regex Matching)
So far each value has been tested against a single pattern. A validation rule in a pipeline needs 4 more tools: alternatives and repeating groups, plus a way to take characters literally and a way to control letter case. Each card below pairs one of them with an input it accepts or rejects. From one pattern to a rule set A large pattern for an email address tells you that a value failed and nothing about why. Splitting the check into small rules inside a CASE expression gives every failing row a