2PC, Outbox, Idempotent Consumers

Concepts covered: Idempotency

Three patterns recur in streaming idempotency engineering: two-phase commit, transactional outbox, and idempotent consumers. Each addresses a specific source of duplicates. Each has a cost that constrains where it applies. A senior engineer reaches for the right one without confusing them, because the pattern that solves consumer-side duplicates does not solve producer-side ones, and vice versa. Naming each precisely is the prerequisite for combining them correctly.

Two-Phase Commit Across Systems

Two-phase commit is the protocol for atomically committing a transaction across two or more systems. Phase one: the coordinator asks each participant to prepare and durably stage the change. Phase two: if all participants prepared successfully, the coordinator tells each to commit; otherwise it tells each to abort. The protocol guarantees that all participants reach the same outcome, all committing or all aborting. Flink and Kafka Streams use two-phase commit between the engine's checkpoint and the sink, which is how engine-level exactly-once is implemented when it works.

Phase 1 (PREPARE):
  Coordinator -> Sink:    'Stage these writes; respond OK or ABORT'
  Sink         -> Coordinator: 'OK' (writes are durable but invisible)

Phase 2 (COMMIT or ABORT):
  If all OK:
    Coordinator -> Sink: 'Commit; make the writes visible'
  Otherwise:
    Coordinator -> Sink: 'Abort; discard the staged writes'

The coordinator's decision is durable;
a crash mid-phase-2 is recoverable because the decision was logged.

Where Two-Phase Commit Breaks Down

The protocol assumes every participant supports the prepare-then-commit handshake and that the coordinator can durably log its decision. Modern systems often satisfy both: Kafka and Flink both implement participants for the protocol, and the coordinator's log lives in a fault-tolerant state store. Older systems often do not. An external HTTP API with no transaction concept cannot participate. A NoSQL store without transaction support cannot participate. When any participant cannot prepare-then-commit, two-phase commit is unavailable and the architecture falls back to either effectively-once (idempotent consumer plus at-least-once) or weaker guarantees.

The Transactional Outbox Pattern

The transactional outbox addresses the dual-write problem: an application that needs to write to a database and publish an event to a queue cannot do both atomically because there is no transaction that spans the two. The pattern is to write only to the database. The transaction includes a row in an outbox table alongside the business write. A separate process tails the outbox table and publishes its rows to the queue. The application gets atomicity (the database's transaction); the queue gets delivery (the outbox tailer). Crashes between the two stages are recoverable because the outbox row is durable, and the tailer can retry until the publish succeeds.

-- Transactional outbox: business write and outbox write in one transaction
-- A separate publisher tails the outbox and emits to Kafka at-least-once
SELECT
  event_id,
  aggregate_id,
  event_type,
  payload,
  published_at IS NULL AS pending
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 10

The pattern guarantees that an event is published if and only if the database write committed. It does not guarantee that the event is published exactly once; the tailer is at-least-once, so consumers must dedup on event_id. The pattern combines with idempotent consumers to produce effectively-once across the application boundary, which is the strongest end-to-end property typically achievable in event-driven systems.

The Idempotent Consumer

An idempotent consumer is one that produces the same end state regardless of how many times it sees the same event. The implementation is usually a deduplication step keyed on a producer-stamped event_id. Before processing, the consumer checks whether the event_id has been seen; if so, it skips. After processing, the consumer records the event_id as processed. The dedup state lives in a fast key-value store (Redis, DynamoDB) or in the destination database itself. The consumer is now safe under at-least-once delivery: duplicates are recognized and dropped.

# An idempotent consumer
def handle(event):
    if dedup_store.has_seen(event.event_id):
        return  # Already processed; safe to skip
    
    with destination.transaction():
        apply_event(event)
        dedup_store.mark_seen(event.event_id, ttl_days=7)

# The transaction ensures that apply_event and mark_seen happen together.
# A crash between them retries the event; the second attempt finds
# nothing in dedup_store and processes again, which is safe because
# apply_event itself is idempotent on the destination key.

How the Three Patterns Combine

A System That Uses All Three

A real high-stakes streaming system often uses all three patterns. The application uses transactional outbox to publish events atomically with its database writes. The streaming engine uses two-phase commit to write to its internal state and a Kafka sink atomically. The downstream consumer uses idempotent consumption keyed on event_id to absorb any duplicates that escape either pattern. The combination is effectively-once at every boundary, which is the strongest guarantee a multi-system pipeline can practically offer. No single pattern is sufficient; the combination is what produces the system-level property.

Naive Dual-Write

• App writes to database AND publishes to Kafka in same handler
• No atomicity; crash between writes leaks one or the other
• Duplicates and lost events both possible
• The single most common production bug in event-driven systems

Outbox + Idempotent Consumer

• App writes to database; outbox row is part of the same transaction
• Tailer publishes outbox rows to Kafka with at-least-once
• Downstream consumer dedups on event_id
• Effectively-once across the application boundary

Do

• Use the transactional outbox for any service that writes to its database and publishes events
• Implement idempotent consumers on every cross-system event handler; the broker is at-least-once by default
• Reserve two-phase commit for engines that natively support it; do not roll a custom 2PC

Avoid

• Use the dual-write anti-pattern; it breaks under any failure that crosses the two writes
• Assume Kafka exactly-once covers writes to a non-Kafka sink; it does not
• Skip the dedup state TTL; an unbounded dedup store eventually becomes the bottleneck

About this lesson

This section is part of the Idempotent Reruns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.