Stream Guarantees
This one comes out of your own answer. You describe a streaming pipeline, mention exactly-once in passing, and the interviewer says: 'Walk me through the consumer crashing after it has processed 3 events but only committed 2.' The rubric separates candidates who know the 3 guarantee names from candidates who can point at the line where the guarantee is decided.
Commit before processing, or after
A consumer reads from 1 partition of the log at an offset, processes the event, and at some point tells the log it is done by committing that offset. The guarantee follows from whether the commit happens before or after the processing. Commit first and a crash in between loses the event, because the log believes it was handled. Process first and a crash in between replays the event, because the log believes it was never handled. The log stores bytes and offsets and promises neither outcome on its own.
| Offset | Event | Processed before crash | Committed before crash | Replayed after restart |
|---|---|---|---|---|
| 0 | play A | yes | yes | no |
| 1 | play B | yes | yes | no |
| 2 | play C | yes | no | yes, a duplicate |
| 3 | play D | no | no | yes |
| 4 | play E | no | no | yes |
Offset 2 is the event that counts twice. The consumer wrote play C to the sink, crashed before committing offset 2, and on restart the log handed back offset 2 as the next unread position. In a plays table that is 1 extra row; in a payments table it is a double charge. At-least-once is the accurate name for this pipeline, and most production streams run exactly here, with deduplication downstream doing the work the word 'exactly' implies.
| Guarantee | Commit timing | After the crash above | Typical use |
|---|---|---|---|
| At-most-once | before processing | play C lost, D and E processed once | metrics where a gap is cheaper than a repeat |
| At-least-once | after processing | play C processed twice | most pipelines, paired with deduplication |
| Exactly-once processing | atomically with the write, or idempotent sink | play C visible once in the sink | billing, inventory, ledgers |
Exactly-once is a property of the whole path, and the phrase interviewers want is 'exactly-once processing, built from at-least-once delivery plus a sink that tolerates repeats.' One way to get that tolerance is for the sink to accept the offset and the output in 1 transaction, so a crash rolls back both together. The other is an idempotent write, meaning repeating it leaves the same single row, usually because the sink keys on event_id and the second copy overwrites or is rejected. Either way, the log alone cannot deliver it; the sink has to participate, and saying so is the difference between hand-waving and an answer.
The numbers behind deduplication
Deduplication needs a key and a window, and the window is the tradeoff to name. The sink keeps the set of keys it has seen for some retention, and a duplicate arriving after that retention passes through. A longer window costs memory or a wider lookup on every write; a shorter one admits late duplicates. Size it from the consumer side and the producer side. On the consumer side, the input is the longest outage you intend to recover from without manual cleanup, because a consumer restarting after a long stall replays every event it processed but never committed, and those copies arrive as late as the outage was long. On the producer side, it is the retry horizon: a producer that retries a send after a timeout can write the same event to the log twice before any consumer runs. A team whose on-call restores any stalled consumer within 20 hours, with producers retrying for at most 10 minutes, adds a few hours of margin to that sum to set the retention. Producer-side sequence numbers or a stable event id minted on the client give every copy the same key, which is what lets the window catch it.
- 24h
- 1 partition
- 2
SELECT
event_id,
COUNT(*) AS copies
FROM plays
WHERE event_date = DATE '2026-03-14'
GROUP BY event_id
HAVING COUNT(*) > 1The ordering follow-up comes next. The log orders events within 1 partition and makes no promise across partitions, so a cancel and the order it cancels can arrive in either sequence if they hash to different partitions. The answer is to key both by the order id so they share a partition, and to accept that this concentrates all of 1 busy customer's traffic on 1 consumer. Naming that cost shows you have chosen a key under pressure before.
- Do
- Say which side of processing the commit happens on before naming the guarantee
- Mint the event id on the producer so retries carry the same key
- Name the dedup window and what a duplicate older than it does
- Key related events together when their order matters
- Avoid
- Claim exactly-once without describing the sink's part in it
- Treat the log's ordering as global
- Commit offsets before the write has been acknowledged
- Forget the producer retry as a duplicate source
About this lesson
This section is part of the How Data Moves: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.