Replay Infrastructure

Concepts covered: Stream Processing

Replay is the streaming-world equivalent of backfill: reprocessing events from a known offset or timestamp to correct downstream state. It's harder than batch backfill because there's no clean partition to overwrite, and easier because the source is often retained in a log that supports random access. Designing for replay requires 3 pieces of infrastructure: a source log retained long enough to cover the replay window, offsets or timestamps a job can seek to, and consumers that can see the same event twice without counting it twice. If the log has already expired the events you need, for instance, replay turns into a manual recovery operation instead of a routine job.

What replay is for

The source retention requirement

Replay assumes the source still has the events to replay. Kafka retains messages by time (default 7 days, often configured to 30 or more) or by total size. Kinesis retains messages for up to 365 days. Pub/Sub has a configurable retention window. The retention window is the limit of how far back replay can reach, so a pipeline that needs to support 6-month replays must run on infrastructure with 6-month retention. Retention costs storage proportional to throughput times retention, and that cost is usually worth paying because it keeps replay a routine operation.

Addressable Positions

Replay requires the ability to start a new consumer (or rewind an existing one) to a specific position in the source. Kafka exposes consumer offsets per partition; replaying from offset N on partition P is a documented operation. Iceberg, Delta Lake, and Hudi expose snapshot IDs and timestamps as addressable points in their tables; reading from snapshot S provides a stable point-in-time view. The position can be a numeric offset, a timestamp, a transaction LSN, or a snapshot ID, but it has to exist as a first-class concept the source supports. Sources that don't expose addressable positions can't be replayed in any meaningful sense.

# Replay from a known Kafka offset
from kafka import KafkaConsumer, TopicPartition

consumer = KafkaConsumer(
    bootstrap_servers='kafka:9092',
    group_id='replay-2026-04-25',  # New group ID prevents committing over production offsets
    enable_auto_commit=False,
)

partition = TopicPartition('orders', 0)
consumer.assign([partition])
consumer.seek(partition, 1450000)  # Resume from the offset where the bug started

for event in consumer:
    handle_idempotently(event)
    if event.offset >= 1500000:
        break  # Replay window complete

The replay reads from offset 1,450,000 to 1,500,000, applies idempotent processing, and stops. The consumer group ID is new, so the production consumer's offsets are unaffected. The downstream destination dedups on event_id, so the replay doesn't produce double-applied effects. The whole operation is bounded, isolated, and reversible if it goes wrong, which is what makes it operationally safe.

Table format time travel

Modern table formats (Iceberg, Delta Lake, Hudi) provide time travel as a first-class feature. Each commit to the table produces a snapshot identified by an ID and a timestamp. A query can address any retained snapshot directly. Replay against a table-format source becomes 'read the table as of snapshot S' or 'read the changes between snapshots S1 and S2'; the reads are stable, addressable, and free of the eventual-consistency artifacts that plague raw object storage. Time travel turns replay from a streaming-only concept into one that applies to any data lake table that uses a modern format.

-- Iceberg time travel: read the table as it was at a past snapshot
SELECT order_id, amount, status
FROM orders
FOR TIMESTAMP AS OF '2026-04-25 12:00:00'
WHERE order_date = '2026-04-25'

The query reads the orders table as it existed at noon on April 25, regardless of how the table has changed since. Snapshot retention determines how far back the query can reach. The mechanism is the same as Kafka offsets in spirit: an addressable position the source exposes as a first-class operation. Time travel makes batch replay against table-format sources as ergonomic as streaming replay against Kafka.

Idempotent Downstreams: The Third Requirement

A retained source and addressable positions are useless if the downstream is not idempotent. Replay re-emits events the downstream has already processed, and a non-idempotent downstream double-counts every replayed event. The third requirement is that every consumer in the pipeline can absorb replays without producing duplicate effects. UPSERT-keyed destinations satisfy this naturally. Aggregations stored as full overwrites by partition satisfy it. Side effects (sending an email, charging a card, calling an external API) generally do not, and replay through those steps requires either skipping them or guarding them with their own dedup logic.

No replay infrastructure

• Source retention is whatever the broker defaults to
• No way to start a consumer at a specific past position
• Downstreams accumulate state non-idempotently
• A bug-fix replay is a multi-week reconstruction project

Replay-ready architecture

• Source retention is sized for the longest expected replay window
• Consumers can be started at any retained offset or snapshot
• Downstreams are idempotent; replay produces correct end state
• A bug-fix replay is a one-day operation with bounded impact

30 days Typical Kafka retention for replay-ready production deployments: Bug detection plus root-cause plus replay execution rarely fits in the default 7-day retention

• Retained Source: Kafka topic retention; Iceberg/Delta snapshot retention. Sized to the longest expected bug-detection-to-replay window.
• Addressable Position: Kafka offsets, Iceberg snapshot IDs, Delta version numbers. The source exposes the position as a first-class concept.
• Idempotent Downstream: UPSERT keyed on event_id, partition overwrite by date, dedup table. The destination absorbs replays without producing duplicate effects.

Do

• Size source retention to cover the longest realistic bug-detection-to-replay window, not the median
• Use new consumer group IDs for replay so production offsets aren't disturbed
• Run replay on isolated compute capacity to protect production throughput

Avoid

• Replay through a pipeline whose downstream side effects aren't idempotent; the email storm isn't theoretical
• Replay over a window large enough to cause source-side rate limits or back-pressure on the broker
• Treat replay as a rare operation; design for it as a feature, not as an emergency

About this lesson

This section is part of the Idempotent Reruns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.