Idempotency in Streaming Is Harder
Concepts covered: Idempotency
Batch idempotency rests on a clean boundary: the partition. The pipeline owns a unit of work, the unit corresponds to a slice of the destination, and the slice can be replaced atomically. Streaming has no equivalent. Events arrive continuously; the destination is being written to continuously; there is no obvious moment at which to draw a boundary and say 'the work for this window is now complete and can be replaced.' Streaming idempotency exists, but someone has to engineer it, and that engineering is significantly more complex than the batch case.
Why the partition trick does not apply
| Property | Batch | Streaming |
|---|---|---|
| Boundary of one run | A clean partition (date, hour, fifteen-minute window) | No clean boundary; processing is continuous |
| Atomic write per run | INSERT OVERWRITE PARTITION; CREATE OR REPLACE | Each event write is independent; partition-level atomicity does not exist |
| Retry semantics | Rerun the partition; previous output replaced | Reprocess from an offset; previous output may already be downstream |
| Backfill | Loop over date partitions; each replaces its slice | Replay from a known offset; downstream must absorb duplicates |
The at-least-once default
Most streaming infrastructure defaults to at-least-once delivery. Kafka commits offsets after the consumer signals it has processed a batch, which means a consumer that crashes between processing and committing will reprocess the same events on restart. The same is true of Kinesis, Pub/Sub, and most queue systems. At-least-once is the easy guarantee to provide because it requires only that the broker not lose messages; the broker isn't responsible for whether the consumer applies them once or twice. Idempotency on top of at-least-once is the work that turns the broker's easy promise into a system-level guarantee.
# A naive Kafka consumer that is not idempotent
# Simulating a crash between processing and offset commit
counter = {'page_views': 0}
events = [{'page_id': 'home', 'event_id': 'evt_1'}, {'page_id': 'home', 'event_id': 'evt_2'}]
offset_committed = -1
# First attempt: process evt_1, crash before commit
counter['page_views'] += 1 # processed
# offset_committed never updated; simulated crash
# Restart: consumer resumes from last committed offset (-1), so evt_1 reprocesses
counter['page_views'] += 1 # re-processed evt_1
offset_committed = 0
counter['page_views'] += 1 # processed evt_2
offset_committed = 1
print(f'Counter after at-least-once redelivery: {counter}')
print(f'Expected count for 2 unique events: 2')
print(f'Actual count: {counter["page_views"]} (one duplicate)')The 2 operations (state update and offset commit) aren't atomic. A crash between them produces a duplicate update on the next run. The fix is to make either the update idempotent (so the duplicate is harmless) or the pair atomic (so the crash never produces a half-state), and neither is trivial.
3 sources of streaming duplicates
The boundary problem
Even when the engine guarantees exactly-once writes to its own state store (Flink's checkpointed state, Kafka Streams' internal topics), the boundary where state crosses into a different system is the place duplicates leak. Writing to a Postgres table, calling an external API, sending a message to another Kafka cluster: each is a boundary where the engine's guarantee ends and the next system's guarantee begins. The end-to-end guarantee is only as strong as the weakest of those boundaries.
| Boundary | Engine Guarantee | End-to-End Guarantee |
|---|---|---|
| Flink internal state -> Flink internal state | Exactly-once via checkpoints | Exactly-once (single engine) |
| Flink -> Kafka sink | Exactly-once via two-phase commit (when configured) | Exactly-once if downstream consumer reads the committed offsets |
| Flink -> Postgres | Idempotent UPSERT or two-phase commit | Exactly-once only if the sink itself is idempotent |
| Flink -> external HTTP API | Engine cannot offer exactly-once; the API is unaware of the transaction | At-least-once at best; the API must be idempotent on its side |
Where streaming idempotency is easier
Streaming idempotency isn't always hard. Aggregations whose output is naturally a function of the entire input (a sum, a count, a max) can use stateful windowing with periodic checkpoints; the engine restores state on recovery and reprocesses only the events after the checkpoint. UPSERT-friendly destinations (key-value stores like Redis or DynamoDB, transactional databases with primary keys) absorb at-least-once events naturally because the second write is a no-op. Stateless transforms whose output is deterministic in the input (filter, project, enrich with reference data) are trivially idempotent. The hard cases are stateful transforms that emit to non-idempotent sinks, which is where exactly-once advertising and engineering reality diverge.
- Easy streaming idempotency
- Stateless filter, project, enrich operations
- Aggregations with checkpointed state and idempotent sink
- UPSERT-friendly destination keyed by event ID
- Append-only log destination with deduplication on read
- Hard streaming idempotency
- Stateful aggregations writing to a non-idempotent sink
- Multi-system transactions (database AND queue AND external API)
- Side effects with no idempotency token (sending email, charging a card)
- Joins across 2 streams with different progress rates
- Streaming has no clean partition to overwrite, so you have to build idempotency into the pipeline yourself.
- At-least-once is the default, and an idempotent consumer turns it into effectively-once.
- End-to-end guarantees are bounded by the weakest link, which is almost always the cross-system boundary.
About this lesson
This section is part of the Idempotent Reruns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.