"What Is Your Delivery Guarantee?"
Concepts covered: Idempotency
What They're Really Testing
The hidden rubric: can this candidate match a business requirement to the right delivery guarantee and defend the cost tradeoff? Candidates who say 'exactly-once for everything' reveal they do not understand the performance cost. Candidates who say 'it depends on the business cost of a duplicate vs a loss' demonstrate the judgment that separates L5 from L6 on the scorecard.
The Three Guarantees
| Semantic | Guarantee | Mechanism | Use When |
|---|---|---|---|
| At-most-once | 0 or 1 delivery. May lose messages. | Commit offset BEFORE processing. If crash after commit, message is lost. | Metrics sampling, non-critical logs, best-effort signals |
| At-least-once | 1+ deliveries. May duplicate. | Commit offset AFTER processing. If crash after process but before commit, message reprocessed. | Most data pipelines. Downstream dedup handles duplicates. |
| Exactly-once | Exactly 1 processing. No loss, no duplicates. | Idempotent producer + transactional consumer + read_committed isolation. | Financial transactions, payment processing, billing. |
The Unlock
The mental model: delivery semantics are about where you commit the offset relative to processing. Commit before processing = at-most-once (if you crash, you skip the message). Commit after processing = at-least-once (if you crash, you reprocess). Commit atomically with processing = exactly-once (but at a cost).
The 60-Second Framework
- Ask: 'What is the business cost of a duplicate?' If catastrophic (double charge), exactly-once.
- Ask: 'What is the business cost of a loss?' If catastrophic (missing audit record), exactly-once.
- If neither is catastrophic: 'At-least-once with idempotent consumers. The consumer's MERGE handles duplicates.'
- State the mechanism: 'Offset committed after processing. Duplicates handled by MERGE on event_id.'
Why Companies Care
At Stripe, payment events use exactly-once because a double-processed payment charge means a customer is billed twice. At Netflix, playback telemetry uses at-least-once because a duplicate play event is corrected by downstream dedup at negligible cost. At Uber, ride fare calculations moved from at-least-once to exactly-once after duplicate fare charges caused a customer trust incident that made the news.
The strongest opening: 'The right delivery semantic depends on the business cost of a duplicate versus the business cost of a loss. For financial data, a duplicate charge is catastrophic, so exactly-once. For clickstream, a duplicate click is corrected by dedup at query time, so at-least-once is sufficient and much cheaper.'
About this lesson
This section is part of the Delivery Semantics: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.