A streaming pipeline serves a real-time dashboard from minute-bucketed revenue

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A streaming pipeline serves a real-time dashboard from minute-bucketed revenue. An audit-grade consumer needs eventually-correct numbers for any closed day, even when long-tail late events arrive after the streaming budget closes. The section's design is a reconciliation pass: a daily batch job that reads the canonical source of truth over a closed window, recomputes the aggregation, atomically overwrites the streaming output's partitions for that window, and emits a delta metric. Both pipelines write to the same target; the latest writer wins per partition. Design the reconciliation by adding an orchestrator that schedules the pass with explicit cadence and window in its name, a batch transform whose name states it reads the canonical source and overwrites by partition atomically, and a delta-metric destination that logs the streaming-vs-reconciliation gap.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.