Exactly-Once and Its Cost
Concepts covered: Idempotency
Exactly-once semantics (EOS) is the most misunderstood concept in streaming. Candidates either dismiss it as impossible or claim it is the only acceptable guarantee. Neither is correct. EOS is achievable within Kafka, with specific mechanisms, at a specific cost.
Kafka's Three-Layer EOS
- Layer 1: The broker assigns each producer a Producer ID (PID) and tracks sequence numbers per partition. If a producer retries a send, the broker detects the duplicate by PID + sequence number and silently drops it. No duplicate writes to Kafka. Enabled by enable.idempotence=true.
- Layer 2: The producer wraps multiple writes in a transaction. Either all writes to all partitions succeed (commit) or none do (abort). This enables atomic consume-transform-produce: read from input topic, transform, write to output topic, commit consumer offset, all in one transaction.
- Layer 3: Consumers set isolation.level=read_committed. They only see messages from committed transactions. Aborted transactions are invisible. This prevents consumers from processing data that was later rolled back.
The Cost of EOS
| Dimension | At-Least-Once | Exactly-Once | Impact |
|---|---|---|---|
| Throughput | Baseline | 15-20% lower | Transaction coordination adds round-trips to the broker |
| Latency | Baseline | Higher (transaction commit) | Each transaction has a commit phase before data is visible |
| Complexity | Simple consumer loop | Transaction lifecycle management | Begin, send, commit offsets, commit transaction, handle abort |
| Debugging | Straightforward offset tracking | Transaction state, PID tracking | Failed transactions require understanding the transaction coordinator |
The Follow-Up Trap
- Follow-Up: Only within Kafka. If the consumer writes to an external system (database, S3), the external write is not part of the Kafka transaction. Strong answer: 'Kafka EOS is exactly-once within the Kafka ecosystem. For external sinks, I need idempotent writes at the sink to achieve effectively-once end-to-end.'
- Follow-Up: Strong answer: 'When the 15-20% throughput cost is unacceptable, or when at-least-once + dedup achieves the same business outcome. For clickstream at 1M events/sec, the EOS overhead would require 20% more Kafka brokers. At-least-once with MERGE dedup is cheaper and operationally simpler.'
About this lesson
This section is part of the Delivery Semantics: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.