Fencing Zombie Producers
Concepts covered: Exactly-once processing
The fare writer runs under an orchestrator that restarts anything that stops answering health checks. Process A owns transactional.id=fare-writer-7, holds producer id 4001 at epoch 0, and has a transaction open with the fare for trip-9001 already at offset 40 of fares-2. Then A stalls for 40 seconds: a garbage collection pause, or a network link that drops. The orchestrator stops hearing from A and starts process B from the same config. Now 2 processes each believe they are fare-writer-7, and A will wake up with sends still queued.
B calls init_transactions(). The coordinator looks up fare-writer-7, finds producer id 4001 at epoch 0 with a transaction ongoing, bumps the epoch to 1, hands B the pair (4001, 1), and aborts A's open transaction: an ABORT marker goes to every partition A had registered. The coordinator's log now says epoch 1 is current, and each partition leader learns the new epoch from the marker it appended.
| Clock | Who | Does | Result |
|---|---|---|---|
| 0 s | A | `init_transactions()` | producer id 4001, epoch 0 |
| 1.0 s | A | `produce()` trip-9001 to `fares-2` | offset 40, epoch 0 |
| 1.2 s | A | stalls | nothing leaves A for 40 s |
| 30 s | orchestrator | starts B with the same config | 2 processes claim `fare-writer-7` |
| 31 s | B | `init_transactions()` | epoch 1; ABORT marker at `fares-2` offset 41 |
| 32 s | B | `produce()` trip-9001, `commit_transaction()` | record at 42, COMMIT marker at 43 |
| 41 s | A | wakes, `produce()` trip-9002 to `fares-2` | rejected: epoch 0 is older than 1, `ProducerFencedException` |
| 41 s | A | `commit_transaction()` | same exception; the only call left is `close()` |
Every produce request carries the producer id and the epoch. The leader of fares-2 already keeps per-producer state for idempotence, the last epoch and sequence number it accepted for 4001, and the marker at offset 41 moved that epoch to 1. A's request at 41 seconds arrives with epoch 0. The leader rejects it as stale, and the client raises ProducerFencedException, which is fatal: the producer can no longer retry, open a new transaction or commit. Trip-9002 from A never enters the log, and a reader of committed records sees trip-9001 exactly once, at offset 42, because offset 40 sits under an ABORT marker.
The id names a shard of work, not a process
Fencing depends on 1 rule: at any moment, 1 live producer per transactional.id. The rule holds only if the id describes something there is 1 of. A service that runs 8 replicas and gives every replica transactional.id=fare-writer breaks it: each replica's init_transactions() bumps the epoch past the other 7, every replica is fenced on its next send, and the service writes nothing. The scheme that works gives each unit of ownership its own id. With 12 shards of trips, the ids are fare-writer-0 through fare-writer-11, and whichever process owns shard 7 opens its producer with fare-writer-7. On a restart the new owner takes the id, the coordinator bumps the epoch, and the old owner, if it is alive anywhere, is fenced on its next request.
- Stable id per shard
- fare-writer-7 belongs to shard 7, whoever runs it
- a restart bumps the epoch and fences the old owner
- the old owner's open transaction is aborted at once
- 2 live owners of 1 shard fence each other, which is the alarm you want
- Fresh id per start
- fare-writer-7-a3f9, a random suffix per start, never matches a previous run
- nothing is ever fenced, so a stalled old process keeps writing
- the previous run's open transaction waits out transaction.timeout.ms
- __transaction_state accumulates an entry per start
An idempotent producer without a transactional.id gets none of this. It asks for a producer id with no name attached, receives a fresh one at epoch 0, and the broker deduplicates its retries against that id alone. 2 instances of it write freely into the same partition. Fencing is a property of the transactional id, because the id is the only thing 2 processes can be made to share.
About this lesson
This section is part of the Producers: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.