Semantics for a Business Need
Concepts covered: Idempotency
A single event stream usually feeds several consumers, and each needs its own guarantee. Picking 1 guarantee for "the pipeline" overpays on the dashboards or underprotects the money. Decide per sink, from 4 questions.
- What does 1 duplicate cost at this sink, in money, trust or wrong decisions?
- What does 1 lost record cost, and would anyone notice it was missing?
- Can the effect be made idempotent with a key minted at the origin?
- Can the source replay for longer than your worst recovery takes?
The 4 questions also order your interview time: spend it on the sinks where both a duplicate and a loss are expensive, because those need the costliest mechanism.
A stream with 4 consumers
Take a payments company's charge events at 20,000 per second, read by 4 consumers, each at its own committed position. Assign each a guarantee from the 4 questions before reading the table.
- charge_events (queue, Partitioned log); parallelism: 200 partitions
- ledger_writer (transform, Consumer); on error: fail closed; idempotency: rows + offsets in 1 txn
- fraud_features (transform, Stream job); idempotency: distinct-count aggregates
- receipt_sender (transform, Consumer); idempotency: send log keyed by payment_id
- live_dashboard (transform, Consumer); alerting: hourly reconciliation
- ledger (storage, Ledger table)
- feature_store (storage, Feature table)
- email_provider (consumer, External API)
- dashboard_tiles (consumer, Dashboard)
- charge_events -> ledger_writer
- charge_events -> fraud_features
- charge_events -> receipt_sender
- charge_events -> live_dashboard
- ledger_writer -> ledger
- fraud_features -> feature_store
- receipt_sender -> email_provider
- live_dashboard -> dashboard_tiles
- Source: charge_events
- Consumers: ledger_writer, fraud_features, receipt_sender, live_dashboard
- Sinks: ledger, feature_store, email_provider, dashboard_tiles
| Consumer | Duplicate cost | Loss cost | Guarantee | Mechanism |
|---|---|---|---|---|
| Ledger writer | A double charge on the books | Missing revenue and a failed audit | Exactly-once | Ledger rows and offsets in 1 transaction |
| Fraud features | None, for distinct-count features | A card missed by a velocity check | At-least-once | Replay into aggregates that absorb repeats |
| Receipt sender | A second email to the customer | No receipt, then a support ticket | At-least-once with dedup | Send log keyed by payment_id, plus the provider's idempotency key |
| Live dashboard | A tile briefly high | A tile briefly low | At-most-once | Commit first, reconcile hourly against the ledger |
Look at the receipt row. A duplicate email and a missing receipt both cost something, and the missing receipt costs more because it becomes a support ticket. Work out that ordering for each sink and the guarantee name follows from it.
Structure of a complete answer
- Guarantee per sink: Each consumer's guarantee, with the business cost that justifies it.
- Mechanism: The order of commit and effect, and the key or transaction that enforces that order.
- Failures handled: How the design handles a retry and a late record, and what happens when the schema changes.
- The number: Whatever the guarantee costs, sized from throughput: the duplicates or losses per crash, or else the dedup state or visibility latency it needs.
- What it still misses: The edge the guarantee leaves uncovered, and how you detect it: a reconciliation, a lag alert, a dead-letter owner.
Retry, late data and schema change in the ledger
| Failure | What the ledger writer does | Result |
|---|---|---|
| Producer retries pay_8812 after a lost acknowledgment | The second copy hits insert-if-absent on payment_id and is skipped inside the transaction | A single ledger row |
| A charge from a terminal offline for 3 hours | Rows key on payment_id and book by event_time, with no watermark to drop the event | The row lands in the correct day, and that day's total is restated |
| Producer adds a currency_code column with a default | Old and new records both parse, and payment_id is unchanged | No rekeying, no skipped records |
| The amount column changes type to string | Parsing fails, so the writer stops and alerts instead of committing past the record | Lag grows; nothing lost, nothing doubled |
The last row fails closed, and you should defend that choice aloud. A halted ledger writer shows up in 1 lag alert and recovers without loss once the schema is fixed, provided the outage stays inside the 7-day retention. A writer that skips unparseable records keeps its lag at zero while revenue disappears.
Payouts are money: exactly-once, through a transactional write with offsets or a payout table keyed on trip ID. Surge pricing reads demand over the last few minutes. A missed or repeated trip moves its multiplier by a fraction, so at-least-once on the replay the job already has is enough; freshness matters more than either guarantee. The rating prompt is a human-facing side effect where a duplicate annoys and a miss costs little, so at-most-once fits, unless the product team tracks prompt coverage, in which case a keyed send log gives at-least-once with dedup.
- Do
- Mint the event ID where the event is born and carry it to every sink
- Size duplicates per crash from throughput times commit interval
- Fail closed on unparseable records and alert on lag
- Reconcile source and sink counts for any sink that accepts loss
- Set sink transaction timeouts above worst checkpoint plus recovery time
- Avoid
- Quote a broker setting as the pipeline's guarantee
- Derive dedup keys from a hash of the payload
- Pick 1 guarantee for every consumer of a stream
- Let consumer lag approach log retention without a page
- Assume exactly-once covers an external API call, whether it sends an email or fires a webhook
About this lesson
This section is part of the Delivery Semantics: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.