Idempotency and Backfill: Advanced
A payment processor at scale ran a streaming aggregator that summed daily revenue across all merchants. The system advertised exactly-once semantics. One Sunday afternoon a Kafka broker hard-crashed and the consumer group rebalanced. The aggregator emitted a small percentage of duplicate sums in the seconds around the rebalance. The duplicates were within the engine's exactly-once guarantee for sink writes, but they violated the merchant-facing API's contract that revenue figures were monotonic. Two days of regulatory reports had to be reissued. The post-incident finding was that exactly-once at the engine level was not exactly-once at the system level, and the gap had been papered over with vendor marketing language. This lesson is about the engineering reality behind idempotency in streaming systems: where the guarantees actually exist, where they shade into effectively-once, and what infrastructure is required to make replay safe at the boundary between exactly-once-mattering domains (financial close) and at-least-once-fine domains (page view counters).
Idempotency in Streaming Is Harder
Recognize why streaming has no clean partition boundary and identify which streaming patterns are naturally idempotent versus which require explicit engineering.
Batch idempotency rests on a clean boundary: the partition. The pipeline owns a unit of work, the unit corresponds to a slice of the destination, and the slice can be replaced atomically. Streaming has no equivalent. Events arrive continuously; the destination is being written to continuously; there is no obvious moment at which to draw a boundary and say 'the work for this window is now complete and can be replaced.' Streaming idempotency exists, but it is engineered, not inherent, and the engineering is significantly more complex than the batch case.
Why the Partition Trick Does Not Apply
| Property | Batch | Streaming |
|---|---|---|
| Boundary of one run | A clean partition (date, hour, fifteen-minute window) | No clean boundary; processing is continuous |
| Atomic write per run | INSERT OVERWRITE PARTITION; CREATE OR REPLACE | Each event write is independent; partition-level atomicity does not exist |
| Retry semantics | Rerun the partition; previous output replaced | Reprocess from an offset; previous output may already be downstream |
| Backfill | Loop over date partitions; each replaces its slice | Replay from a known offset; downstream must absorb duplicates |
The At-Least-Once Default
Most streaming infrastructure defaults to at-least-once delivery. Kafka commits offsets after the consumer signals it has processed a batch, which means a consumer that crashes between processing and committing will reprocess the same events on restart. The same is true of Kinesis, Pub/Sub, and most queue systems. At-least-once is the easy guarantee to provide because it requires only that the broker not lose messages; the broker is not responsible for whether the consumer applies them once or twice. Idempotency on top of at-least-once is the work that turns the broker's easy promise into a system-level guarantee.
The two operations (state update and offset commit) are not atomic. A crash between them produces a duplicate update on the next run. The fix is to make either the update idempotent (so the duplicate is harmless) or the pair atomic (so the crash never produces a half-state). Both fixes are non-trivial. Both are the subject of this lesson.
Three Sources of Streaming Duplicates
Where the duplicates come from:
- ▸Producer retries: a producer that does not get an ack republishes the same event
- ▸Consumer crashes between processing and offset commit
- ▸Consumer group rebalances: partition ownership shifts mid-batch and the new owner starts from the last committed offset
The Boundary Problem
Even when the engine guarantees exactly-once writes to its own state store (Flink's checkpointed state, Kafka Streams' internal topics), the boundary where state crosses into a different system is the place duplicates leak. Writing to a Postgres table, calling an external API, sending a message to another Kafka cluster: each is a boundary where the engine's guarantee ends and the next system's guarantee begins. The end-to-end guarantee is the weakest link in the chain, not the strongest.
| Boundary | Engine Guarantee | End-to-End Guarantee |
|---|---|---|
| Flink internal state -> Flink internal state | Exactly-once via checkpoints | Exactly-once (single engine) |
| Flink -> Kafka sink | Exactly-once via two-phase commit (when configured) | Exactly-once if downstream consumer reads the committed offsets |
| Flink -> Postgres | Idempotent UPSERT or two-phase commit | Exactly-once only if the sink itself is idempotent |
| Flink -> external HTTP API | Engine cannot offer exactly-once; the API is unaware of the transaction | At-least-once at best; the API must be idempotent on its side |
Where Streaming Idempotency Is Easier
Streaming idempotency is not always hard. Aggregations whose output is naturally a function of the entire input (a sum, a count, a max) can use stateful windowing with periodic checkpoints; the engine restores state on recovery and reprocesses only the events after the checkpoint. UPSERT-friendly destinations (key-value stores like Redis or DynamoDB, transactional databases with primary keys) absorb at-least-once events naturally because the second write is a no-op. Stateless transforms whose output is deterministic in the input (filter, project, enrich with reference data) are trivially idempotent. The hard cases are stateful transforms that emit to non-idempotent sinks, which is where exactly-once advertising and engineering reality diverge.
✓Easy Streaming Idempotency
- Stateless filter, project, enrich operations
- Aggregations with checkpointed state and idempotent sink
- UPSERT-friendly destination keyed by event ID
- Append-only log destination with deduplication on read
•Hard Streaming Idempotency
- Stateful aggregations writing to a non-idempotent sink
- Multi-system transactions (database AND queue AND external API)
- Side effects with no idempotency token (sending email, charging a card)
- Joins across two streams with different progress rates
Streaming has no clean partition boundary; idempotency is engineered rather than inherent.
At-least-once is the default; idempotency on the consumer side turns it into effective once.
End-to-end guarantees are bounded by the weakest link, which is almost always the cross-system boundary.
TIP
When evaluating a streaming system's idempotency, draw the boundary lines explicitly. Inside a single engine, exactly-once is often achievable. Across boundaries, the conversation is about effectively-once and which side carries the deduplication burden.
Exactly-Once vs Effectively-Once
Distinguish exactly-once from effectively-once, audit a vendor claim against the actual deployment, and choose the right pattern for the system in question.
Exactly-once is one of the most loaded phrases in streaming. Vendor marketing has used it for so long that the engineering meaning has eroded. The honest framing: exactly-once is achievable inside a closed system where the engine controls every read, write, and offset commit. End-to-end exactly-once across systems is generally not achievable; what gets advertised under that name is more precisely called effectively-once, which is at-least-once delivery combined with idempotent consumers. The distinction is not pedantry; it determines what failure modes are possible and where the deduplication work lives.
Three Delivery Guarantees
| Guarantee | Meaning | Typical Where |
|---|---|---|
| At-most-once | Each event is delivered zero or one times | Fire-and-forget logging; metrics where some loss is tolerable |
| At-least-once | Each event is delivered one or more times | Default for Kafka, Kinesis, Pub/Sub; strong guarantee that no event is lost |
| Exactly-once | Each event is processed once and only once at the boundary in question | Achievable inside a single engine; harder across boundaries |
What Exactly-Once Actually Guarantees
Inside a single Flink job with checkpointed state and a transactional sink, exactly-once means that the effect of each input event on the engine's output is applied once even across failures. The mechanism is two-phase commit: the engine writes to the sink in a transaction, the transaction is committed only when the corresponding checkpoint completes, and a failure rolls back any uncommitted output. Kafka Streams provides a similar guarantee when both source and sink are Kafka topics participating in the same transaction. The guarantee is real, but it is bounded to the systems participating in the transaction.
What Effectively-Once Means
Effectively-once is the system-level outcome where each event affects the final state once, achieved by combining at-least-once delivery with idempotent processing. The broker may deliver an event multiple times. The consumer may apply it multiple times. The destination is shaped so that the second application has no effect (UPSERT on a primary key, dedup table keyed on event ID, monotonic counter that ignores duplicates). The end-state property is the same as exactly-once. The implementation is fundamentally different. The implementation difference matters because effectively-once requires the destination to participate; exactly-once requires the engine to participate. The honest characterization for most production deployments is 'effectively-once,' not the stricter engine-level claim.
Exactly-OnceEffectively-OnceAt-Least-Once
Exactly-Once
Engine-controlled
Two-phase commit between the engine and the sink. Bounded to systems that participate in the transaction. The engine bears the complexity.
Effectively-Once
Consumer-controlled
At-least-once plus idempotent processing on the consumer side. Works across system boundaries because the destination owns deduplication.
At-Least-Once
Broker-controlled
Default for Kafka, Kinesis, Pub/Sub. Guarantees no event loss. Duplicates are possible and must be handled downstream.
Reading Vendor Claims
When a vendor advertises exactly-once, the right question is 'across which boundary.' Kafka exactly-once is exactly-once across Kafka topics within a Kafka transaction; it does not cover writes to a non-Kafka sink. Flink exactly-once is exactly-once for the engine's state; it covers external sinks only when those sinks are configured to participate in two-phase commit, which is a non-trivial configuration that many teams skip. Spark Structured Streaming exactly-once depends on the sink supporting idempotent writes, typically via a checkpoint-aware connector. The marketing word is the same; the engineering reality is per-deployment.
Questions to ask of an exactly-once claim:
- ▸Across which boundaries does the guarantee hold?
- ▸What sinks are supported, and is the production sink one of them?
- ▸What configuration is required to enable the guarantee, and is it on?
- ▸What is the latency cost (two-phase commit adds 100ms or more per checkpoint)?
- ▸What is the failure mode if the guarantee is violated; is it visible or silent?
When Effectively-Once Is the Right Choice
Effectively-once is often the better engineering choice even when exactly-once is available. The complexity is lower because the destination owns deduplication. The latency is lower because there is no two-phase commit handshake. The system is more portable because it does not depend on engine-specific transaction semantics. The cost is the engineering work to make destinations idempotent, but that work is local to the destination and reusable across producers. A system with idempotent destinations and at-least-once delivery is operationally simpler than a system with exactly-once delivery to mutable destinations.
The MERGE inserts each event_id once. A duplicate delivery of the same event_id matches and does nothing. The destination is idempotent, the broker is at-least-once, and the system as a whole is effectively-once. No two-phase commit is required. The engineering investment is in maintaining the dedup index on event_id, which is small relative to the operational simplification.
•Exactly-Once via 2PC
- Engine-level guarantee tied to specific connectors
- 100ms or more added latency per checkpoint
- Configuration is fragile; downgrades are possible without alerts
- Bounded to systems that support the engine's transaction protocol
✓Effectively-Once via Idempotent Sink
- Destination owns deduplication via UPSERT or dedup keys
- No transaction overhead; latency unchanged
- Broker delivers at-least-once; configuration is the default
- Works across any combination of producer and consumer systems
Exactly-once is a property of a closed system. Effectively-once is a property of an open system that has been engineered to absorb duplicates. The honest characterization of most production deployments labeled exactly-once is closer to 'effectively-once,' and naming it precisely makes the failure modes legible.
✓Do
- Default to at-least-once delivery plus idempotent sinks for cross-system pipelines
- Reserve engine-level exactly-once for closed-system patterns where the engine owns both source and sink
- Audit exactly-once claims against the actual sink configuration in production
✗Don't
- Believe the marketing word without naming the boundary it applies across
- Configure two-phase commit on a sink that does not support it; the guarantee silently downgrades
- Skip dedup logic on the destination because the broker promises exactly-once; downstream changes invalidate the assumption
2PC, Outbox, Idempotent Consumers
Apply two-phase commit, transactional outbox, and idempotent consumers as the building blocks for streaming idempotency, and combine them across the boundaries of a real architecture.
Three patterns recur in streaming idempotency engineering: two-phase commit, transactional outbox, and idempotent consumers. Each addresses a specific source of duplicates. Each has a cost that constrains where it applies. A senior engineer reaches for the right one without confusing them, because the pattern that solves consumer-side duplicates does not solve producer-side ones, and vice versa. Naming each precisely is the prerequisite for combining them correctly.
Two-Phase Commit Across Systems
Two-phase commit is the protocol for atomically committing a transaction across two or more systems. Phase one: the coordinator asks each participant to prepare and durably stage the change. Phase two: if all participants prepared successfully, the coordinator tells each to commit; otherwise it tells each to abort. The protocol guarantees that all participants reach the same outcome, all committing or all aborting. Flink and Kafka Streams use two-phase commit between the engine's checkpoint and the sink, which is how engine-level exactly-once is implemented when it works.
Where Two-Phase Commit Breaks Down
The protocol assumes every participant supports the prepare-then-commit handshake and that the coordinator can durably log its decision. Modern systems often satisfy both: Kafka and Flink both implement participants for the protocol, and the coordinator's log lives in a fault-tolerant state store. Older systems often do not. An external HTTP API with no transaction concept cannot participate. A NoSQL store without transaction support cannot participate. When any participant cannot prepare-then-commit, two-phase commit is unavailable and the architecture falls back to either effectively-once (idempotent consumer plus at-least-once) or weaker guarantees.
The Transactional Outbox Pattern
The transactional outbox addresses the dual-write problem: an application that needs to write to a database and publish an event to a queue cannot do both atomically because there is no transaction that spans the two. The pattern is to write only to the database. The transaction includes a row in an outbox table alongside the business write. A separate process tails the outbox table and publishes its rows to the queue. The application gets atomicity (the database's transaction); the queue gets delivery (the outbox tailer). Crashes between the two stages are recoverable because the outbox row is durable, and the tailer can retry until the publish succeeds.
The pattern guarantees that an event is published if and only if the database write committed. It does not guarantee that the event is published exactly once; the tailer is at-least-once, so consumers must dedup on event_id. The pattern combines with idempotent consumers to produce effectively-once across the application boundary, which is the strongest end-to-end property typically achievable in event-driven systems.
The Idempotent Consumer
An idempotent consumer is one that produces the same end state regardless of how many times it sees the same event. The implementation is usually a deduplication step keyed on a producer-stamped event_id. Before processing, the consumer checks whether the event_id has been seen; if so, it skips. After processing, the consumer records the event_id as processed. The dedup state lives in a fast key-value store (Redis, DynamoDB) or in the destination database itself. The consumer is now safe under at-least-once delivery: duplicates are recognized and dropped.
How the Three Patterns Combine
| Pattern | Solves | Cost |
|---|---|---|
| Two-phase commit | Engine-to-sink atomicity in a closed system | Latency (100ms+ per checkpoint); requires participant support |
| Transactional outbox | Atomicity between an application's database write and an event publish | Outbox table, tailer process, dedup-on-event-id downstream |
| Idempotent consumer | Duplicates from at-least-once delivery on the consumer side | Dedup state store; lookup cost per event |
A System That Uses All Three
A real high-stakes streaming system often uses all three patterns. The application uses transactional outbox to publish events atomically with its database writes. The streaming engine uses two-phase commit to write to its internal state and a Kafka sink atomically. The downstream consumer uses idempotent consumption keyed on event_id to absorb any duplicates that escape either pattern. The combination is effectively-once at every boundary, which is the strongest guarantee a multi-system pipeline can practically offer. No single pattern is sufficient; the combination is what produces the system-level property.
•Naive Dual-Write
- App writes to database AND publishes to Kafka in same handler
- No atomicity; crash between writes leaks one or the other
- Duplicates and lost events both possible
- The single most common production bug in event-driven systems
✓Outbox + Idempotent Consumer
- App writes to database; outbox row is part of the same transaction
- Tailer publishes outbox rows to Kafka with at-least-once
- Downstream consumer dedups on event_id
- Effectively-once across the application boundary
Pattern selection rule:
- ▸Atomic engine-to-sink in a closed system: two-phase commit
- ▸Atomic application-database-to-queue in any system: transactional outbox
- ▸Tolerating duplicates on the consumer side: idempotent consumer
- ▸All three may apply in different parts of the same architecture
✓Do
- Use the transactional outbox for any service that writes to its database and publishes events
- Implement idempotent consumers on every cross-system event handler; the broker is at-least-once by default
- Reserve two-phase commit for engines that natively support it; do not roll a custom 2PC
✗Don't
- Use the dual-write anti-pattern; it breaks under any failure that crosses the two writes
- Assume Kafka exactly-once covers writes to a non-Kafka sink; it does not
- Skip the dedup state TTL; an unbounded dedup store eventually becomes the bottleneck
TIP
When a system claims exactly-once, identify which of the three patterns is providing it. If the answer is none of them, the claim is a misnomer and the system is silently at-least-once.
Replay Infrastructure
Design replay infrastructure that combines retained sources, addressable positions, and idempotent downstreams, and reason about the cost of each.
Replay is the streaming-world equivalent of backfill. It is the act of reprocessing events from a known offset or timestamp to correct downstream state. Replay is harder than batch backfill because there is no clean partition to overwrite, and easier because the source is often retained in a log that supports random access. Designing for replay requires three pieces of infrastructure: a retained source, addressable positions, and idempotent downstream consumers. Without all three, replay is a manual recovery operation. With them, replay is a feature.
What Replay Is For
| Replay Trigger | Frequency | Typical Range |
|---|---|---|
| Bug in the consumer logic; reprocess to correct downstream state | Several times per year per active stream | Hours to weeks |
| New consumer being onboarded; needs history | Per consumer launch | Whatever retention allows; typically days to months |
| Source schema correction; downstream needs to reprocess | Occasional | Days to weeks |
| Disaster recovery; a downstream system needs to be rebuilt | Rare but inevitable | Full retained history |
The Source Retention Requirement
Replay assumes the source still has the events to replay. Kafka retains messages by time (default seven days, often configured to thirty or more) or by total size. Kinesis retains messages for up to three hundred sixty five days. Pub/Sub has a configurable retention window. The retention window is the limit of how far back replay can reach. A pipeline that needs to support six-month replays must run on infrastructure with six-month retention. Retention is not free; it costs storage proportional to throughput times retention. The cost is real but usually justified by the operational value of replay being a feature rather than an emergency.
Addressable Positions
Replay requires the ability to start a new consumer (or rewind an existing one) to a specific position in the source. Kafka exposes consumer offsets per partition; replaying from offset N on partition P is a documented operation. Iceberg, Delta Lake, and Hudi expose snapshot IDs and timestamps as addressable points in their tables; reading from snapshot S provides a stable point-in-time view. The position can be a numeric offset, a timestamp, a transaction LSN, or a snapshot ID, but it has to exist as a first-class concept the source supports. Sources that do not expose addressable positions cannot be replayed in any meaningful sense.
The replay reads from offset 1,450,000 to 1,500,000, applies idempotent processing, and stops. The consumer group ID is new, so the production consumer's offsets are unaffected. The downstream destination dedups on event_id, so the replay does not produce double-applied effects. The whole operation is bounded, isolated, and reversible if it goes wrong, which is what makes it operationally safe.
Table Format Time Travel
Modern table formats (Iceberg, Delta Lake, Hudi) provide time travel as a first-class feature. Each commit to the table produces a snapshot identified by an ID and a timestamp. A query can address any retained snapshot directly. Replay against a table-format source becomes 'read the table as of snapshot S' or 'read the changes between snapshots S1 and S2'; the reads are stable, addressable, and free of the eventual-consistency artifacts that plague raw object storage. Time travel turns replay from a streaming-only concept into one that applies to any data lake table that uses a modern format.
The query reads the orders table as it existed at noon on April 25, regardless of how the table has changed since. Snapshot retention determines how far back the query can reach. The mechanism is the same as Kafka offsets in spirit: an addressable position the source exposes as a first-class operation. Time travel makes batch replay against table-format sources as ergonomic as streaming replay against Kafka.
Idempotent Downstreams: The Third Requirement
A retained source and addressable positions are useless if the downstream is not idempotent. Replay re-emits events the downstream has already processed, and a non-idempotent downstream double-counts every replayed event. The third requirement is that every consumer in the pipeline can absorb replays without producing duplicate effects. UPSERT-keyed destinations satisfy this naturally. Aggregations stored as full overwrites by partition satisfy it. Side effects (sending an email, charging a card, calling an external API) generally do not, and replay through those steps requires either skipping them or guarding them with their own dedup logic.
The replay infrastructure checklist:
- ▸Source retains history long enough to cover the longest expected replay (Kafka retention, table format snapshot retention)
- ▸Source exposes addressable positions (Kafka offsets, snapshot IDs, timestamps)
- ▸Every downstream consumer is idempotent on the event ID or the business key
- ▸Replay can run on isolated compute that does not interfere with production processing
- ▸Side effects (emails, charges, external API calls) are guarded against re-emission during replay
•No Replay Infrastructure
- Source retention is whatever the broker defaults to
- No way to start a consumer at a specific past position
- Downstreams accumulate state non-idempotently
- A bug-fix replay is a multi-week reconstruction project
✓Replay-Ready Architecture
- Source retention is sized for the longest expected replay window
- Consumers can be started at any retained offset or snapshot
- Downstreams are idempotent; replay produces correct end state
- A bug-fix replay is a one-day operation with bounded impact
Retained SourceAddressable PositionIdempotent Downstream
Retained Source
History to read from
Kafka topic retention; Iceberg/Delta snapshot retention. Sized to the longest expected bug-detection-to-replay window.
Addressable Position
Where to resume
Kafka offsets, Iceberg snapshot IDs, Delta version numbers. The source exposes the position as a first-class concept.
Idempotent Downstream
Safe to replay into
UPSERT keyed on event_id, partition overwrite by date, dedup table. The destination absorbs replays without producing duplicate effects.
✓Do
- Size source retention to cover the longest realistic bug-detection-to-replay window, not the median
- Use new consumer group IDs for replay so production offsets are not disturbed
- Run replay on isolated compute capacity to protect production throughput
✗Don't
- Replay through a pipeline whose downstream side effects are not idempotent; the email storm is not theoretical
- Replay over a window large enough to cause source-side rate limits or back-pressure on the broker
- Treat replay as a rare operation; design for it as a feature, not as an emergency
TIP
When a streaming system has no replay story, the on-call rotation is one bug away from a multi-week recovery. Investing in retention and idempotent downstreams early is cheaper than the first incident that exposes their absence.
Two Streaming Aggregators
Design two streaming aggregators with opposite guarantee requirements, justify each architectural choice against the use case, and articulate when exactly-once is over-engineering.
The patterns become concrete on real workloads. Two streaming aggregators sit at opposite ends of the idempotency-cost spectrum. The first is a financial close aggregator that produces daily revenue numbers used in regulatory reporting; exactly-once is a correctness requirement, and the cost of getting it wrong is real money and real regulatory exposure. The second is a page view counter that powers a real-time engagement dashboard; at-least-once is sufficient, the dashboard tolerates noise, and the cost of full exactly-once would be wasted. The same architectural team would build the two systems differently. The exercise below walks through both.
Aggregator A: Financial Close (Exactly-Once Matters)
The system aggregates payment events from a Kafka topic into a daily revenue table that finance uses for the monthly close. Outputs go into a regulatory report that is filed with the SEC. Duplicates are not noise; they are misstated revenue, which carries legal liability. The system must produce identical output across replays and recoveries. The architecture is engineered top to bottom for exactly-once at the system level.
| Component | Choice | Why |
|---|---|---|
| Source | Kafka with 90-day retention | Replay window must cover full quarter detection-to-correction cycle |
| Engine | Flink with checkpointed state and two-phase commit sink | Native exactly-once across engine state and the destination |
| Destination | Iceberg table with primary key on event_id | Idempotent UPSERT absorbs any duplicates that escape 2PC |
| Downstream | Daily snapshot report keyed on revenue_date | Reports are reproducible from the table at any past snapshot |
| Replay | Bounded window with new consumer group; idempotent destination dedupes | Bug-fix replays produce identical output to original runs |
The two-phase commit binds the engine's checkpoint to the sink's commit. A failure mid-checkpoint rolls back any uncommitted writes. The Iceberg destination is also idempotent on event_id, which provides defense in depth: even if the two-phase commit fails in some unforeseen way, duplicates are absorbed by the UPSERT. The retention window is generous because financial close bugs are sometimes detected weeks or months after the fact, and the replay window must cover detection-to-correction.
Aggregator B: Page View Counter (At-Least-Once Is Fine)
The system aggregates page view events into a per-page counter shown on a real-time engagement dashboard. The dashboard refreshes every five seconds. A few duplicate counts during a Kafka rebalance produce a few extra page views in the count for a few seconds. The dashboard's audience is product managers and growth analysts who care about trends, not absolute precision. The architecture is engineered for throughput and operational simplicity, not for engine-level exactly-once.
| Component | Choice | Why |
|---|---|---|
| Source | Kafka with 7-day retention | Replay over a few days is the realistic upper bound; longer is overkill |
| Engine | Kafka Streams with at-least-once and idempotent counter | Operational simplicity; counters are inherently noisy on the millisecond scale |
| Destination | Redis counter keyed by (page_id, hour_bucket) | INCR is fast; small over-counts during failures are acceptable |
| Downstream | Dashboard reading the Redis counter every 5 seconds | Display layer; not used for accounting or billing |
| Replay | Not engineered; if the counter drifts, recompute from raw events as a batch job | Cost of full replay infrastructure exceeds the value of perfect counts |
The page view counter is intentionally simple. The INCR operation is non-idempotent in the strict sense: a duplicate event increments the counter twice. The system absorbs the imprecision because the use case tolerates it. A daily batch reconciliation pass replaces the streaming counter with the correct count from the immutable raw events; small streaming-side drifts are corrected in batch without the streaming pipeline needing exactly-once semantics. The pattern is sometimes called a Lambda architecture for this specific reason: streaming for low latency, batch for correctness, both writing to the same dashboard with batch winning at the daily boundary.
✓Aggregator A: Financial Close
- Exactly-once at the system level is a hard requirement
- Two-phase commit between Flink and Iceberg
- 90-day retention; bug-fix replays must reach back a quarter
- UPSERT keyed on event_id provides defense in depth
- Cost of architectural complexity is justified by regulatory exposure
✓Aggregator B: Page View Counter
- At-least-once with bounded over-counting is acceptable
- Kafka Streams with INCR to Redis counter
- 7-day retention; dashboard never looks back further
- Daily batch reconciliation corrects any streaming drift
- Cost of architectural simplicity is justified by use case tolerance
The Decision Framework
The choice between the two architectures is not a tools decision; it is a use-case decision. Three questions drive it. First, what is the cost of a duplicate effect? In financial close, the cost is regulatory exposure and reissued reports. In a page view counter, the cost is a few percent noise on a dashboard. Second, what is the latency budget? Two-phase commit adds 100ms or more per checkpoint, which a financial close pipeline can absorb but a real-time dashboard cannot. Third, what is the engineering cost ceiling? Exactly-once architectures take real engineering investment to design, build, and operate; the investment pays back only when the use case demands it.
When exactly-once at the system level is the right choice:
- ▸Outputs are used for accounting, billing, or regulatory reporting
- ▸Duplicates produce real money or real legal exposure, not just noise
- ▸Latency budget can absorb 2PC overhead (typically 100ms+ per checkpoint)
- ▸Team has the operational maturity to run a 2PC system in production
When at-least-once is the right choice:
- ▸Outputs power dashboards, analytics, or anomaly detection where small noise is fine
- ▸Bounded over-counting is correctable via a daily batch reconciliation pass
- ▸Latency budget is tight (sub-100ms end-to-end)
- ▸Team prefers operational simplicity over engine-level guarantees
The Closing Principle
Idempotency in streaming is not one technique; it is a portfolio of techniques chosen against the use case. The financial close aggregator and the page view counter share zero infrastructure choices and yet are both correct architectures for their respective problems. The senior engineering move is not to apply exactly-once everywhere reflexively; it is to ask which guarantee the use case actually requires and to invest the architectural complexity proportional to the answer. Over-engineering is its own bug, and exactly-once where at-least-once would suffice is one of its more expensive forms.
Match the guarantee to the cost of a duplicate effect; regulatory exposure earns 2PC, dashboard noise does not.
Two-phase commit adds latency and operational complexity; reserve it for use cases that need it.
Lambda-style batch reconciliation lets streaming approximations live alongside batch correctness on the same dashboard.
The concise statement of the principle: pick the weakest guarantee that the use case can tolerate, then engineer the system to deliver it reliably. Stronger guarantees are not free; they trade architectural complexity for properties the use case may not need.
✓Do
- Match the idempotency guarantee to the use case; do not apply exactly-once where at-least-once suffices
- Use Lambda-style batch reconciliation when streaming approximations are acceptable in the short window
- Document the chosen guarantee in the pipeline contract; future maintainers need to know what was promised
✗Don't
- Default to exactly-once everywhere; the operational cost is real
- Believe at-least-once is fine when the downstream actually requires precision
- Mix guarantees within a single pipeline without explicit boundary documentation; the chain is as strong as the weakest link
TIP
When designing a new streaming pipeline, write the guarantee requirement in the contract before choosing tools. The requirement determines the tool, not the other way around.
❯❯❯PUTTING IT ALL TOGETHER
> A payment processor at scale runs both kinds of streaming aggregators in production. The financial close aggregator was blamed last week for a duplicate-revenue incident during a Kafka rebalance; regulatory reports for two days had to be reissued. The page view counter has been running with at-least-once for two years without incident. The principal engineer is asked: 'How should the next generation of streaming pipelines at this company be designed so the financial close architecture is as solid as the page view counter is operable, without over-engineering either one?'
Step one: name the property each pipeline requires. The financial close needs effectively-once at the system level; the page view counter needs at-least-once with bounded drift. The naming determines every downstream choice. (Builds on the four-roles model from Lesson 1: source, transform, storage, consumer.)
Step two: design the financial close around three patterns: transactional outbox at the producer, two-phase commit from Flink to Iceberg as the engine-to-sink boundary, and idempotent UPSERT on event_id as defense in depth. The combination produces effectively-once across every boundary, which is the strongest practical end-to-end property.
Step three: design the page view counter for operational simplicity. Kafka Streams at-least-once into a Redis counter, with a daily batch reconciliation pass that recomputes from raw events. The architecture borrows from the layered shape (raw layer plus serving layer) introduced in Lesson 1 intermediate; streaming and batch coexist on the same source of truth.
Step four: invest in replay infrastructure regardless of the chosen guarantee. 90-day Kafka retention for the financial close, 7-day for the page view counter; addressable positions via offsets and Iceberg snapshots; idempotent downstreams keyed on event_id. Replay turns from an emergency operation into a feature. (Extends the orchestrator backfill pattern from Lesson 4 advanced into the streaming domain.)
Step five: store both pipelines in lakehouse storage with time travel (Lesson 3 advanced). The financial close uses Iceberg snapshots for reproducible quarterly reports; the page view counter uses Iceberg time travel as the canonical record against which the streaming approximation is reconciled. Storage choice is the foundation that makes replay and reconciliation work.
Step six: document each pipeline's guarantee in a contract that names producer, consumer, freshness SLA, idempotency strategy, and replay window. The contract converts the architectural decision into a binding artifact downstream consumers can rely on. (Extends the partition-overwrite and DELETE-then-INSERT idempotency contracts from Lesson 5 beginner and intermediate.)
KEY TAKEAWAYS
Streaming has no clean partition boundary: idempotency is engineered via two-phase commit, transactional outbox, and idempotent consumers, not inherited from a partition overwrite.
Exactly-once is closed-system; effectively-once is open-system: engine-level guarantees stop at the boundary the engine controls; cross-system guarantees come from idempotent destinations plus at-least-once delivery.
The transactional outbox solves the dual-write problem: applications write to a database and an outbox in one transaction; a tailer publishes outbox rows to the queue. Combined with idempotent consumers it produces effectively-once across the application boundary.
Replay infrastructure has three components: retained source (Kafka retention, table format snapshots), addressable positions (offsets, snapshot IDs), and idempotent downstreams. All three are required for replay to be a feature rather than an emergency.
Match the guarantee to the use case: exactly-once for financial close and regulatory reporting; at-least-once with batch reconciliation for dashboards and operational counters. Over-engineering is its own bug.
Streaming idempotency, exactly-once claims, and replay infrastructure separate marketing from engineering
- Category
- Pipeline Architecture
- Difficulty
- advanced
- Duration
- 38 minutes
- Challenges
- 0 hands-on challenges
Topics covered: Idempotency in Streaming Is Harder, Exactly-Once vs Effectively-Once, 2PC, Outbox, Idempotent Consumers, Replay Infrastructure, Two Streaming Aggregators
Lesson Sections
- Idempotency in Streaming Is Harder (concepts: paStreamingIdempotency, paAtLeastOnce)
Batch idempotency rests on a clean boundary: the partition. The pipeline owns a unit of work, the unit corresponds to a slice of the destination, and the slice can be replaced atomically. Streaming has no equivalent. Events arrive continuously; the destination is being written to continuously; there is no obvious moment at which to draw a boundary and say 'the work for this window is now complete and can be replaced.' Streaming idempotency exists, but it is engineered, not inherent, and the engi
- Exactly-Once vs Effectively-Once (concepts: paExactlyOnce, paEffectivelyOnce)
Exactly-once is one of the most loaded phrases in streaming. Vendor marketing has used it for so long that the engineering meaning has eroded. The honest framing: exactly-once is achievable inside a closed system where the engine controls every read, write, and offset commit. End-to-end exactly-once across systems is generally not achievable; what gets advertised under that name is more precisely called effectively-once, which is at-least-once delivery combined with idempotent consumers. The dis
- 2PC, Outbox, Idempotent Consumers (concepts: paTwoPhaseCommit, paTransactionalOutbox, paIdempotentConsumer)
Three patterns recur in streaming idempotency engineering: two-phase commit, transactional outbox, and idempotent consumers. Each addresses a specific source of duplicates. Each has a cost that constrains where it applies. A senior engineer reaches for the right one without confusing them, because the pattern that solves consumer-side duplicates does not solve producer-side ones, and vice versa. Naming each precisely is the prerequisite for combining them correctly. Two-Phase Commit Across Syste
- Replay Infrastructure (concepts: paReplay, paTimeTravel)
Replay is the streaming-world equivalent of backfill. It is the act of reprocessing events from a known offset or timestamp to correct downstream state. Replay is harder than batch backfill because there is no clean partition to overwrite, and easier because the source is often retained in a log that supports random access. Designing for replay requires three pieces of infrastructure: a retained source, addressable positions, and idempotent downstream consumers. Without all three, replay is a ma
- Two Streaming Aggregators (concepts: paStreamingAggregatorDesign, paGuaranteeTradeoffs)
The patterns become concrete on real workloads. Two streaming aggregators sit at opposite ends of the idempotency-cost spectrum. The first is a financial close aggregator that produces daily revenue numbers used in regulatory reporting; exactly-once is a correctness requirement, and the cost of getting it wrong is real money and real regulatory exposure. The second is a page view counter that powers a real-time engagement dashboard; at-least-once is sufficient, the dashboard tolerates noise, and