CTEs (Common Table Expressions): Intermediate
Structuring a Multi-CTE Query
Identify when a problem benefits from named intermediate results: multi-step logic, reused subqueries, or readability.
Here is the question that recurs in interviews past the basics. 'You have a transactions table, a customers table, and a refunds table. Compute net monthly revenue per customer segment, with refunds applied, and validate that no segment shows negative net revenue for more than two consecutive months.' The candidate who writes one giant CTE chain that mixes the aggregation, the validation, and the output is producing a query that works but cannot be reviewed. The candidate who builds a CTE pipeline with explicit data quality checkpoints (one CTE that should return zero rows if the data is clean; a final SELECT that fails loudly if it does not) writes a query that is also a test.
- ▸"compute X, then validate that Y"
- ▸"walk a hierarchy and aggregate per level"
- ▸"generate a date range and join actuals onto it"
- ▸"this query is slow; how would you change it"
- ▸"this CTE is referenced three times; what would you do"
The three design questions
- ▸Is this CTE materialized, inlined, or does the engine decide?
- ▸Is recursion the right tool, or is there a non-recursive solution?
- ▸Is this CTE a query-building block, or a data quality gate?
- ▸Each question pushes the candidate from syntax into design
- Writes a CTE chain without thinking about materialization
- Reaches for recursion as the first tool for any hierarchy question
- Treats every CTE as a query-building block; never as an assertion
- Adds CTEs as the query grows without refactoring
- Names the materialization decision out loud (inline vs MATERIALIZED)
- Defends recursive CTE choice with a depth-and-fanout argument
- Uses one CTE per pipeline stage and one CTE per data quality check
- Refactors the chain when it crosses 8-10 CTEs
Why the design layer matters at this level
State the design choice when you write each CTE. 'This CTE is the aggregation step; it produces one row per (customer, month). This next CTE is a data quality check; it returns rows only if any segment shows negative revenue, and the final SELECT should fail loudly if it has rows.' Naming the role of each CTE is the move that proves you treat the query as a design, not a string.
CTE vs Subquery vs Temp Table
Write a clean chain of 2-4 CTEs where each builds on the previous, with meaningful names that tell a story.
When inlining helps and when it hurts
- let the engine inline; predicate pushdown from the outer query helps.
- force MATERIALIZED; the body computes once, the references read from the materialized result.
- let inline; the optimizer may deduplicate via common subexpression elimination.
Engine-specific syntax
- The CTE body is expensive (large aggregate, complex window function)
- The CTE is referenced multiple times in the same query
- You can prove via EXPLAIN that the engine is re-computing on each reference
- On Postgres 12+ where the hint is supported
- The CTE is referenced exactly once (most common case)
- Predicate pushdown from the outer query would speed up the inner query
- The CTE body is cheap (a simple filter or projection)
- On engines without explicit hints; trust the cost-based optimizer
How to tell which the engine chose
Other modern optimizations CTEs do not break
Materialization and the Optimizer Fence
Explain the tradeoffs: CTEs are syntactic (most engines inline them), temp tables materialize, subqueries nest.
The canonical recursive shape
Three common variants
When recursion is the wrong tool
- ▸Running totals: SUM OVER with a frame, not a recursive CTE
- ▸Fixed-depth hierarchies: chained LEFT JOINs, not recursion
- ▸Adjacent-row comparisons: LAG and LEAD over a partition
- ▸Pair finding within a table: self-join with an inequality
- Date spines and series generation (every day in a range, every integer 1 to N)
- Unbounded hierarchy traversal (all reports, all descendants, full subtree)
- Iteration where each step depends on the prior one in non-window ways
- Generating cumulative paths or arrays that window functions cannot produce
- Running totals or rolling sums (SUM OVER with a frame is faster and cleaner)
- Fixed-depth hierarchy walks (two or three LEFT JOINs)
- Sequential value comparisons (LAG and LEAD over a partition)
- Pair finding within a table (self-join with an inequality)
Cycle detection and depth bounds belong in every production CTE
Series generation as a portable date spine
The recursive CTE has one syntactic gotcha across engines: some require the WITH RECURSIVE keyword (Postgres, MySQL 8); some make recursion implicit when the CTE references itself (SQL Server). State the engine you are writing for: 'on Postgres I would write WITH RECURSIVE; on SQL Server WITH alone is sufficient.' Engine-aware writing is the production-experience signal.
Reusing a CTE Across Multiple Branches
Write a recursive CTE for org charts, date spines, and graph traversal, and explain the base + recursive structure.
A technique most candidates have not seen: CTEs as data quality gates. A pipeline query that computes a result is also a query that should verify the result is correct. Write a CTE that should return zero rows when the data is clean. Have the final SELECT either return the violations (for inspection) or raise an error (for pipeline enforcement). This pattern is the conceptual foundation of dbt's tests, Great Expectations checks, and most data-quality frameworks; you can implement it in plain SQL with CTEs.
The assertion CTE pattern
Two consumption modes for the violations CTE
Composability: stacking quality checks
- net revenue should never be negative; rows that are negative are violations.
- values above a sanity threshold may signal a unit error or a data import bug.
- every (segment, month) in the date spine should have a row; missing rows are gaps.
CTE-driven idempotent writes
- Final SELECT returns the answer
- Pipeline framework runs the query and consumes the result
- No validation; downstream consumer trusts the output
- Bugs surface in production when the consumer notices
- Final SELECT returns either the answer or the violations
- Pipeline framework checks the violations CTE before consuming
- Validation is part of the query, not a separate test
- Bugs surface in CI when the violations CTE returns rows
The interviewer often probes this depth indirectly: 'how would you know if this query produced wrong results?' The answer is the violations CTE pattern. The query that produces the data also produces the assertions about the data; if the assertions ever return rows, the consumer knows something is wrong without needing a separate test.
Refactoring Nested Subqueries into CTEs
Practice narrating CTE logic step-by-step, which interviewers use to gauge your communication and clarity of thought.
The 20-CTE monster
The shadow-OOP anti-pattern
- customer_base: SELECT * FROM customers
- customer_with_revenue: add revenue column
- customer_with_orders: add order count
- customer_with_segment: add segment column
- customer_final: SELECT * FROM customer_with_segment
- customer_summary: customer + revenue + order count + segment in one CTE
- monthly_metrics: aggregated metrics in one CTE
- ranked: window function applied to monthly_metrics
- Final SELECT: filter and format
CTE-driven debugging at scale
Real-world failure mode
Refactoring strategies
- ▸Fold sibling enrichments into one CTE if they share the same input
- ▸Promote stable intermediates to views if other queries also need them
- ▸Split into staged queries: one writes a staging table, the other reads
| Situation | Phrasing that flatlines | Phrasing that lands |
|---|---|---|
| Query has 15+ CTEs | "It works." | "This is past the readability threshold for one query. I'd split it into a staging step and a reporting step, or promote the stable intermediates to views." |
| A CTE is referenced three times | "Whatever the engine does." | "On Postgres 12+ I'd add WITH ... AS MATERIALIZED so the body computes once. On other engines I'd write to a temp table." |
| The interviewer asks about recursion | "I'd use a recursive CTE." | "Depends on the depth. Bounded depth gets chained joins; unbounded gets a recursive CTE with cycle detection and a depth bound; high-read at scale gets a materialized closure table." |
| The interviewer asks 'how would you know this is right' | "I'd write tests." | "I'd add a violations CTE inside this query: rows that violate a contract become the result. If the result is empty the data is clean; if not, the pipeline framework fails on it." |
| Each CTE adds one column | "More CTEs is more readable." | "Folding sibling enrichments into one CTE is cleaner. Each CTE should reflect a logical step, not one column." |
The closing summary
> You are in a data engineering interview at an e-commerce company. The interviewer asks: 'Compute net monthly revenue per customer segment with refunds applied, then validate that no segment has negative net revenue. The query needs to run nightly and the team treats data quality violations as a hard failure.'
CTE is inlined by default, so predicate pushdown, partition pruning, projection pruning, and join reordering all still work. The legacy Postgres advice that CTEs are an optimization fence stopped being true in Postgres 12.CTE is referenced multiple times, because the engine recomputes it per reference. Postgres 12 and later take WITH ... AS MATERIALIZED; Snowflake, BigQuery, and SQL Server have no equivalent hint, so materialize into a temp table instead.CTE Scan node under its own name, while an inlined one is substituted into the outer plan and its name never appears.WITH RECURSIVE when depth is unbounded (date spines, hierarchy walks) and not when it is bounded. Running totals belong in SUM() OVER, and a two-level manager chain belongs in two LEFT JOINs. Every production recursive CTE carries a path array for cycle detection and an explicit depth bound.CTEs do not make queries faster; they make your interview answer readable
- Category
- SQL
- Difficulty
- intermediate
- Duration
- 25 minutes
- Challenges
- 0 hands-on challenges
Topics covered: Structuring a Multi-CTE Query, CTE vs Subquery vs Temp Table, Materialization and the Optimizer Fence, Reusing a CTE Across Multiple Branches, Refactoring Nested Subqueries into CTEs
Lesson Sections
- Structuring a Multi-CTE Query (concepts: sqlCte)
The three design questions Three questions sit underneath every design-grade CTE conversation. First: is this CTE materialized, inlined, or does the engine decide? The optimizer's behavior affects whether predicate pushdown works, whether multiple references re-execute, and whether the query plan you see is the plan the engine actually runs. Second: is recursion the right tool here, or am I forcing recursion into a problem that has a non-recursive solution? Third: is this CTE a query-building bl
- CTE vs Subquery vs Temp Table (concepts: sqlCte)
On modern engines, the default behavior for a CTE is to inline it into the surrounding query. The optimizer treats the CTE as if it were a nested subquery in the FROM clause and applies the same optimizations: predicate pushdown, join reordering, common subexpression elimination. Most of the time this is what you want. There is one case where it bites, and one syntactic tool to force the engine's hand. When inlining helps and when it hurts Inlining helps in the common case: a CTE that filters or
- Materialization and the Optimizer Fence (concepts: sqlRecursiveCte)
Recursive CTEs are the only standard SQL syntax for iteration. The engine runs an anchor query, then iterates a recursive step until no new rows are produced. The shape is the same across engines: WITH RECURSIVE name AS (anchor UNION ALL recursive_step). Past the basics, the question is rarely about syntax. It is about when to reach for recursion at all, and when a window function or self-join is cleaner. The canonical recursive shape Three pieces. The anchor produces the starting row. The recur
- Reusing a CTE Across Multiple Branches (concepts: sqlCte)
The assertion CTE pattern Five CTEs. monthly_gross and monthly_refunds compute the aggregates. net_revenue joins them. quality_violations is the gate: any (segment, month) with negative net revenue is a violation. The final SELECT returns the violations. If the data is clean, the result set is empty. If there are violations, the result set is non-empty and the consumer can inspect each one. This is the same pattern dbt uses for its 'singular tests': write a SELECT that returns rows representing
- Refactoring Nested Subqueries into CTEs (concepts: sqlCte)
CTEs are a power tool. The same tool that makes a 50-line query readable can produce a 500-line query that is unreadable for a different reason: too many tiny CTEs, each doing one tiny thing, with the actual logic distributed across the chain. The interviewer is checking whether you recognize the anti-patterns and refactor before they ship to production. The 20-CTE monster If a query has 20 CTEs, something has gone wrong. Either the query is solving a problem that should be decomposed across mul