Gaps and Islands: Intermediate
The ROW_NUMBER Difference Trick
Recognize gaps-and-islands triggers: "consecutive days," "unbroken streaks," "contiguous ranges."
- ▸Gap is a threshold (30 minutes), not strict adjacency
- ▸Row-number difference trick does not apply directly
- ▸LAG plus cumulative SUM is the generalization
- ▸The CASE expression encodes the boundary condition
The two canonical forms
Form 1 is the row-number difference trick. Value column minus row number is constant within a run because both advance in lockstep at every row. Best when 'consecutive' means strict adjacency: integers off by one, dates one day apart, hours one hour apart. Form 2 is LAG plus cumulative SUM. Use LAG to expose each row's predecessor; compute a boolean is_new_run that is 1 when the row starts a new run (predecessor is NULL or the gap exceeds the threshold) and 0 otherwise; cumulative SUM of the boolean produces a run identifier that increments at each boundary. Best when 'consecutive' has a threshold or condition. Both produce equivalent run identifiers; the algebra is the same but expressed differently.
- The sequence column has strict adjacency (off-by-one integers, off-by-one dates)
- The runs are dense (no skipped values within a run)
- Readability matters; the trick is one line of algebra
- Performance matters; one window function vs three for the LAG form
- The gap is a threshold (sessions within N minutes, runs within N days)
- The boundary condition is more complex than "value plus one"
- You need to add conditions per row (status changes, attribute flips)
- The CASE in the boundary expression encodes business logic
What the interviewer is silently scoring
When the interviewer hands you the variant, name it before writing SQL. 'This is gaps and islands with a threshold gap. I'll use the LAG-plus-cumulative-SUM form because the row-number difference does not generalize to threshold gaps.' Two sentences. The interviewer has heard the form choice with a reason; the rubric item for 'can the candidate pick the right tool' is now scored.
Date Gaps vs Integer Gaps
Derive island group IDs by subtracting ROW_NUMBER from the sequence value to produce a constant per island.
The canonical sessionization query
Reading the query top-down
Why three window functions, not one
Why the row-number difference does not work here
The frame clause ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW is mandatory on the cumulative SUM. Without it, the engine defaults to RANGE, which treats tied event_time values as a peer group and assigns them the same cumulative sum. If two events happen at the same nanosecond (rare but possible at scale), RANGE collapses them and the session_id becomes wrong. ROWS is the right default for cumulative sums in this context. The same rule from the running-totals lesson applies here.
Island Aggregates: Length, Start, End
Adapt the technique for date sequences (DATE_DIFF minus ROW_NUMBER) vs integer sequences, handling weekends and holidays.
The status-change variant
What changed in the boundary CASE
Filtering at the right point
Notice the WHERE status = 'active' is in the final SELECT, after the runs CTE. This is critical. If you filter to status = 'active' before computing the run_id, you remove the non-active rows entirely, and the run_id no longer increments when the status was 'paused' between two 'active' stretches. The two separate active runs collapse into one. The filter has to be after the run_id is computed, not before. State this when the interviewer asks: 'I'm filtering after the run_id is computed, because filtering before would collapse adjacent active runs that had a paused stretch between them.'
- Drops non-active rows from the source
- The run_id no longer detects the boundary at status changes
- Two separate active runs merge into one
- Returns inflated run lengths and incorrect run counts
- Run_id correctly increments at every status change
- Each contiguous active stretch gets its own run_id
- Filter keeps only the active runs in the final output
- Run lengths and run counts match the actual data
The general principle: when to filter
The exception to this rule: if you filter the source by an entity (drop a single user, ignore a subscription tier), that filter is safe because it does not affect the boundary detection within any remaining entity. Filters that operate at the same grain as the run boundary are unsafe; filters that operate above the grain (per user, per entity) are safe. Articulating this distinction is what tells the interviewer you have thought about the filter placement, not just copied the pattern.
Handling Ties and Duplicate Timestamps
After identifying islands, compute streak length, start/end boundaries, and filter for streaks exceeding a threshold.
'Only return runs of at least N events'
Computing statistics across the runs
The 'longest session' question, with ties
Grouping by the Computed Island Key
Solve gaps-and-islands using LAG + cumulative SUM for interviewers who want the sessionization variant.
Performance: partition shuffles dominate
Edge case: the empty partition
Edge case: same-timestamp events
- Two events at the same timestamp; engine picks predecessor arbitrarily
- Same query against same data may return different session_ids run to run
- Reproducibility issue; impossible to debug without seeing the underlying order
- Tests pass in dev (no ties) but fail in prod (ties happen)
- ORDER BY event_time, event_id guarantees a deterministic order
- Same data produces the same result on every run
- Tests reproduce in dev because the order is stable
- One extra column in the ORDER BY; no other change needed
The synthesis: two forms, one algebra
- n minus rn is constant in a run because both sequences advance by one per row. Constant changes at every gap.
- boundary flag is 1 at run boundaries, 0 within. Cumulative SUM is constant within a run, increments at boundaries.
- row-number form is shorter for strict adjacency; LAG form generalizes to thresholds and value-change boundaries.
| Variant | Boundary condition | Right form |
|---|---|---|
| Runs of consecutive integers | Strict +1 adjacency | Row-number difference (cleaner) |
| Runs of consecutive days | Strict +1 day | Row-number difference (cleaner) |
| Sessions within 30 minutes | Threshold gap | LAG plus cumulative SUM |
| Status-change runs | Value change (status flip) | LAG plus cumulative SUM |
| Conditional islands (status = 'active') | Value change plus per-row filter | LAG plus cumulative SUM with filter after |
The closing summary
> You are in a data engineering interview at a SaaS analytics company. The interviewer asks: 'Compute user session boundaries from an events table where a session is a sequence of events with no gap longer than thirty minutes. Then return the sessions with at least five events.'
LAG plus a cumulative SUM of a boundary flag instead.CASE expression in the LAG form encodes any boundary condition, which is why it generalizes to thresholds and status flips.status = 'healthy' before the boundary detection silently merges runs separated by rows you removed. Filters above the run grain, such as dropping a whole user, stay safe.ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW on the cumulative sum. The default RANGE frame treats tied timestamps as one peer group and hands them the same running total, which corrupts the run identifier.HAVING, after the GROUP BY on the run key. Add a deterministic tiebreaker such as ORDER BY event_time, event_id so same-timestamp rows pick the same predecessor on every run, and expect the partition shuffle on the entity column to dominate cost at scale.Consecutive sequences break most candidates; the trick is a two-window difference
- Category
- SQL
- Difficulty
- intermediate
- Duration
- 25 minutes
- Challenges
- 0 hands-on challenges
Topics covered: The ROW_NUMBER Difference Trick, Date Gaps vs Integer Gaps, Island Aggregates: Length, Start, End, Handling Ties and Duplicate Timestamps, Grouping by the Computed Island Key
Lesson Sections
- The ROW_NUMBER Difference Trick (concepts: sqlLagLead)
Here is the question that recurs in interviews past the basics. 'Compute user sessions from an events table, where a session is a sequence of events with no gap longer than thirty minutes.' This is gaps and islands with a threshold instead of strict adjacency. The row-number difference trick from the beginner lesson does not apply directly; the gap is not 'value plus one,' it is 'value within some delta.' The LAG-plus-cumulative-SUM form generalizes to this case. Knowing when to reach for it, an
- Date Gaps vs Integer Gaps (concepts: sqlLagLead)
The LAG-plus-cumulative-SUM form is three CTEs instead of two. LAG exposes the predecessor's value. A CASE expression flags rows that start a new run. A cumulative SUM of the flag produces a run identifier. Each CTE is one job; the query reads top-down as the story of the calculation. The canonical sessionization query Reading the query top-down The lagged CTE adds a prev_event_time column via LAG; for the first event per user, prev_event_time is NULL. The flagged CTE adds a boolean is_new_sessi
- Island Aggregates: Length, Start, End (concepts: sqlLagLead)
The other common variant is conditional islands. The runs are not defined by adjacency at all; they are defined by a status or attribute being constant for a stretch of rows. 'Find contiguous periods where a subscription was active.' 'Find stretches where a server was healthy.' 'Find runs of days the stock closed above its 50-day moving average.' These are gaps and islands where 'consecutive' means 'sharing the same value' rather than 'adjacent in the sequence.' The LAG form handles this case cl
- Handling Ties and Duplicate Timestamps (concepts: sqlHaving)
After the run_id is in place, the rest of the query is GROUP BY and aggregation. The interviewer at this level will probe whether you handle per-island thresholds and conditional aggregates correctly. The two most common probes: 'only return runs of at least N events' and 'compute statistics across the runs.' Each one tests where the threshold belongs in the query. 'Only return runs of at least N events' This is a per-island threshold. The filter applies to the aggregate (COUNT, SUM, MAX) after
- Grouping by the Computed Island Key (concepts: sqlLagLead)
The closing escalation at this level covers three areas: performance on large tables, edge cases that ship to production, and the synthesis that ties the two canonical forms together. Hitting all three is what flips the verdict from hire to strong hire on this question. Performance: partition shuffles dominate Both canonical forms partition by the entity column (user_id, subscription_id) and order within the partition. The dominant cost on a distributed engine is the shuffle that co-locates each