# A streaming pipeline serves a real-time dashboard from minute-bucketed revenue Canonical URL: Domain: Pipeline Design ยท Difficulty: medium ## Problem A streaming pipeline serves a real-time dashboard from minute-bucketed revenue. An audit-grade consumer needs eventually-correct numbers for any closed day, even when long-tail late events arrive after the streaming budget closes. The section's design is a reconciliation pass: a daily batch job that reads the canonical source of truth over a closed window, recomputes the aggregation, atomically overwrites the streaming output's partitions for that window, and emits a delta metric. Both pipelines write to the same target; the latest writer wins per partition. Design the reconciliation by adding an orchestrator that schedules the pass with explicit cadence and window in its name, a batch transform whose name states it reads the canonical source and overwrites by partition atomically, and a delta-metric destination that logs the streaming-vs-reconciliation gap. ## Related - [All practice problems](https://datadriven.io/problems) - [Mock interview mode](https://datadriven.io/interview/a-streaming-pipeline-serves-a-real-time-dashboard-from-minut-4e5fa0f9) - [System Design Interview Questions](https://datadriven.io/data-engineering-system-design) - [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep) - [Daily Challenge](https://datadriven.io/daily) --- Source: DataDriven (https://datadriven.io). 100% free data engineering interview prep. Live code execution against Postgres 16, Python 3.11, and Spark sandboxes. No paywall, no premium tier, no signup gate.