Ghosts of Snapshots Past

An easy AI Coding interview practice problem on DataDriven.

Domain
AI Coding
Difficulty
easy
Seniority
L4
Asked in
Netflix

Problem

Yesterday a bad backfill wrote 2M corrupt rows into events_iceberg. We rolled back to the prior snapshot. Now an analyst's AS OF (timestamp) query for a window two days before the rollback is also returning the corrupt rows. Reproduce by running tests/test_time_travel.py. The bug is in how our reader resolves a snapshot for a timestamp; fix it so AS OF queries reflect what was visible at that time.

The past keeps rewriting itself.

Worked solution and explanation

The disguise

This is a graph-lineage query wearing a timestamp costume. read_as_of looks like a plain 'find the latest commit at or before ts' scan, and that is exactly the trap: an Iceberg-style log is a lineage, not a flat timeline. A rollback record does not delete the bad snapshot, it records that the table's live history jumped back to an earlier one. Scan by committed_at alone and you will happily serve snapshot 3, the 2M-row corrupt backfill, to anyone whose AS OF window lands after the backfill but before the rollback.

The buggy resolve
eligible = [s for s in log if s['committed_at'] <= ts]
chosen = eligible[-1]  # latest by time, blind to the rollback

Treats rollback as just another commit.

Why the bug stays hidden

Here is the cruel part: snapshot 4 has the SAME content as snapshot 2, its rollback target. So AS OF anything after 2026-04-28 11:00 returns clean rows and looks correct; AS OF anything before snapshot 3's commit also looks correct. The bug lives in one narrow window, 04-27 10:00 to 04-28 11:00, exactly where the analyst was querying. A test that only checks 'after rollback' passes green while production serves corruption.

Latest by `committed_at`
Sorts every commit by time and takes the last one <= ts. Treats the rollback record as an ordinary commit, so for ts inside the corrupt window it returns snapshot 3.
Live-lineage aware
First derives which snapshots a rollback removed from the lineage, then picks the latest REMAINING snapshot <= ts. Snapshot 3 is never a candidate.

The fix

Step 1: Derive what each rollback removed

For each rollback record, parent_id is the snapshot we rolled back TO. Every non-rollback snapshot committed strictly between that target's time and the rollback's own time is no longer in the live lineage. Here that set is {3}, computed from the log, never hardcoded.

Step 2: Filter, then pick the latest

Keep snapshots with committed_at <= ts AND not in the removed set, then take the one with the greatest committed_at. Use max(..., key=...) rather than eligible[-1]: once you filter, the tail is no longer guaranteed to be the newest.

Step 3: Return early on empty

If nothing is eligible (a ts before any commit), return [], the analyst asked about a time the table did not yet exist.

lake/reader.py, lineage-aware
from datetime import datetime
from lake.metadata import snapshot_log
from lake.storage import read_snapshot


def _rolled_back_ids(log):
    """Snapshot ids a later rollback removed from the live lineage.

    A 'rollback' record's parent_id points at the snapshot we rolled
    back TO, so every snapshot committed after that target and before
    the rollback itself is no longer part of the table's history.
    """
    by_id = {s['snapshot_id']: s for s in log}
    removed = set()
    for s in log:
        if s['operation'] != 'rollback':
            continue
        target_time = by_id[s['parent_id']]['committed_at']
        for other in log:
            if other['operation'] == 'rollback':
                continue
            if target_time < other['committed_at'] < s['committed_at']:
                removed.add(other['snapshot_id'])
    return removed


def read_as_of(table: str, ts: datetime):
    """Return rows visible in `table` at timestamp `ts`."""
    log = snapshot_log(table)
    removed = _rolled_back_ids(log)
    eligible = [
        s for s in log
        if s['committed_at'] <= ts and s['snapshot_id'] not in removed
    ]
    if not eligible:
        return []
    chosen = max(eligible, key=lambda s: s['committed_at'])
    return read_snapshot(table, chosen['snapshot_id'])

Common follow-up questions