Two Streaming Aggregators

Concepts covered: Idempotency

The patterns become concrete on real workloads. Two streaming aggregators sit at opposite ends of the idempotency-cost spectrum. The first is a financial close aggregator that produces daily revenue numbers used in regulatory reporting; exactly-once is a correctness requirement, and the cost of getting it wrong is real money and real regulatory exposure. The second is a page view counter that powers a real-time engagement dashboard; at-least-once is sufficient, the dashboard tolerates noise, and the cost of full exactly-once would be wasted. The same architectural team would build the two systems differently. The exercise below walks through both.

Aggregator A: Financial Close (Exactly-Once Matters)

The system aggregates payment events from a Kafka topic into a daily revenue table that finance uses for the monthly close. Outputs go into a regulatory report that is filed with the SEC. Duplicates are not noise; they are misstated revenue, which carries legal liability. The system must produce identical output across replays and recoveries. The architecture is engineered top to bottom for exactly-once at the system level.

// Flink two-phase commit sink for the financial close aggregator
// (sketch; production code uses Iceberg's exactly-once sink)
DataStream<RevenueEvent> revenue = env
    .addSource(new FlinkKafkaConsumer<>(
        "payments",
        new RevenueDeserializer(),
        kafkaProps))
    .keyBy(RevenueEvent::getMerchantId)
    .window(TumblingEventTimeWindows.of(Time.days(1)))
    .aggregate(new RevenueAggregator());

revenue.addSink(IcebergSink.forRowData(rowData)
    .table(loader)
    .upsert(true)              // Idempotent UPSERT on primary key
    .tableSchema(schema)
    .toTwoPhaseCommitSink());  // Two-phase commit between checkpoint and sink

The two-phase commit binds the engine's checkpoint to the sink's commit. A failure mid-checkpoint rolls back any uncommitted writes. The Iceberg destination is also idempotent on event_id, which provides defense in depth: even if the two-phase commit fails in some unforeseen way, duplicates are absorbed by the UPSERT. The retention window is generous because financial close bugs are sometimes detected weeks or months after the fact, and the replay window must cover detection-to-correction.

Aggregator B: Page View Counter (At-Least-Once Is Fine)

The system aggregates page view events into a per-page counter shown on a real-time engagement dashboard. The dashboard refreshes every five seconds. A few duplicate counts during a Kafka rebalance produce a few extra page views in the count for a few seconds. The dashboard's audience is product managers and growth analysts who care about trends, not absolute precision. The architecture is engineered for throughput and operational simplicity, not for engine-level exactly-once.

# Page view counter consumer
# At-least-once delivery; small over-counts are tolerable
for event in consumer:
    bucket = event.timestamp.replace(minute=0, second=0, microsecond=0)
    redis.hincrby(f"pageviews:{bucket.isoformat()}", event.page_id, 1)
    consumer.commit(event.offset)

# A consumer crash between hincrby and commit may double-count one event.
# The dashboard refreshes every 5 seconds; the over-count is invisible.
# A daily batch job recomputes from raw events to correct any drift.

The page view counter is intentionally simple. The INCR operation is non-idempotent in the strict sense: a duplicate event increments the counter twice. The system absorbs the imprecision because the use case tolerates it. A daily batch reconciliation pass replaces the streaming counter with the correct count from the immutable raw events; small streaming-side drifts are corrected in batch without the streaming pipeline needing exactly-once semantics. The pattern is sometimes called a Lambda architecture for this specific reason: streaming for low latency, batch for correctness, both writing to the same dashboard with batch winning at the daily boundary.

Aggregator A: Financial Close

• Exactly-once at the system level is a hard requirement
• Two-phase commit between Flink and Iceberg
• 90-day retention; bug-fix replays must reach back a quarter
• UPSERT keyed on event_id provides defense in depth
• Cost of architectural complexity is justified by regulatory exposure

Aggregator B: Page View Counter

• At-least-once with bounded over-counting is acceptable
• Kafka Streams with INCR to Redis counter
• 7-day retention; dashboard never looks back further
• Daily batch reconciliation corrects any streaming drift
• Cost of architectural simplicity is justified by use case tolerance

The Decision Framework

The choice between the two architectures is not a tools decision; it is a use-case decision. Three questions drive it. First, what is the cost of a duplicate effect? In financial close, the cost is regulatory exposure and reissued reports. In a page view counter, the cost is a few percent noise on a dashboard. Second, what is the latency budget? Two-phase commit adds 100ms or more per checkpoint, which a financial close pipeline can absorb but a real-time dashboard cannot. Third, what is the engineering cost ceiling? Exactly-once architectures take real engineering investment to design, build, and operate; the investment pays back only when the use case demands it.

The Closing Principle

Idempotency in streaming is not one technique; it is a portfolio of techniques chosen against the use case. The financial close aggregator and the page view counter share zero infrastructure choices and yet are both correct architectures for their respective problems. The senior engineering move is not to apply exactly-once everywhere reflexively; it is to ask which guarantee the use case actually requires and to invest the architectural complexity proportional to the answer. Over-engineering is its own bug, and exactly-once where at-least-once would suffice is one of its more expensive forms.

• Match the guarantee to the cost of a duplicate effect; regulatory exposure earns 2PC, dashboard noise does not.
• Two-phase commit adds latency and operational complexity; reserve it for use cases that need it.
• Lambda-style batch reconciliation lets streaming approximations live alongside batch correctness on the same dashboard.

The concise statement of the principle: pick the weakest guarantee that the use case can tolerate, then engineer the system to deliver it reliably. Stronger guarantees are not free; they trade architectural complexity for properties the use case may not need.

Do

• Match the idempotency guarantee to the use case; do not apply exactly-once where at-least-once suffices
• Use Lambda-style batch reconciliation when streaming approximations are acceptable in the short window
• Document the chosen guarantee in the pipeline contract; future maintainers need to know what was promised

Avoid

• Default to exactly-once everywhere; the operational cost is real
• Believe at-least-once is fine when the downstream actually requires precision
• Mix guarantees within a single pipeline without explicit boundary documentation; the chain is as strong as the weakest link

About this lesson

This section is part of the Idempotent Reruns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.