A streaming pipeline serves a real-time dashboard from minute-bucketed revenue
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
Interview Prompt
A streaming pipeline serves a real-time dashboard from minute-bucketed revenue. An audit-grade consumer needs eventually-correct numbers for any closed day, even when long-tail late events arrive after the streaming budget closes. The section's design is a reconciliation pass: a daily batch job that reads the canonical source of truth over a closed window, recomputes the aggregation, atomically overwrites the streaming output's partitions for that window, and emits a delta metric. Both pipelines write to the same target; the latest writer wins per partition. Design the reconciliation by adding an orchestrator that schedules the pass with explicit cadence and window in its name, a batch transform whose name states it reads the canonical source and overwrites by partition atomically, and a delta-metric destination that logs the streaming-vs-reconciliation gap.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision