Apache Iceberg Debug: Ghosts of Snapshots Past

Yesterday a bad backfill wrote 2M corrupt rows into events_iceberg. We rolled back to the prior snapshot. Now an analyst's AS OF (timestamp) query for a window two days before the rollback is also returning the corrupt rows. Reproduce by running tests/test_time_travel.py. The bug is in how our reader resolves a snapshot for a timestamp; fix it so AS OF queries reflect what was visible at that time.

lake/reader.py

# lake/reader.py
# Thin reader over an Iceberg-style metadata log. Picks a snapshot for a
# given AS OF timestamp and reads it.
#
# Snapshot log shape (lake/metadata.py):
#   [(snapshot_id, committed_at, parent_id, operation), ...]
# operation is 'append', 'overwrite', or 'rollback'.
# A 'rollback' record has parent_id pointing at the snapshot we rolled BACK TO.

from datetime import datetime
from lake.metadata import snapshot_log
from lake.storage import read_snapshot


def read_as_of(table: str, ts: datetime):
    """Return rows visible in `table` at timestamp `ts`."""
    log = snapshot_log(table)
    # Pick the latest snapshot whose committed_at <= ts.
    eligible = [s for s in log if s['committed_at'] <= ts]
    if not eligible:
        return []
    chosen = eligible[-1]
    return read_snapshot(table, chosen['snapshot_id'])

# lake/reader.py
# Thin reader over an Iceberg-style metadata log. Picks a snapshot for a
# given AS OF timestamp and reads it.
#
# Snapshot log shape (lake/metadata.py):
#   [(snapshot_id, committed_at, parent_id, operation), ...]
# operation is 'append', 'overwrite', or 'rollback'.
# A 'rollback' record has parent_id pointing at the snapshot we rolled BACK TO.

from datetime import datetime
from lake.metadata import snapshot_log
from lake.storage import read_snapshot


def read_as_of(table: str, ts: datetime):
    """Return rows visible in `table` at timestamp `ts`."""
    log = snapshot_log(table)
    # Pick the latest snapshot whose committed_at <= ts.
    eligible = [s for s in log if s['committed_at'] <= ts]
    if not eligible:
        return []
    chosen = eligible[-1]
    return read_snapshot(table, chosen['snapshot_id'])

Active Now|Data Engineer (L4)|||3.8k Attempts|981 Solves|