Every Device, Every Impression
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L6
Problem
How would you design a pipeline a revenue analyst can trust for two numbers fed by the same playback telemetry from millions of devices: ad billing, settled with partners every day, and content analytics, which tolerates late, approximate counts?
Every ad seen. Every second watched. Real-time.
Business requirements
- Ad billing partners reconcile against our impression counts; if we don't match within tolerance, payment is blocked.
- Some events arrive twice from device retries; ad billing can't count the same impression twice, and a missed one is lost revenue.
- An ad that doesn't run to completion within the threshold counts as a partial impression billed at half credit; this happens often enough that we can't apply the rule by hand.
- Devices in low-connectivity locations buffer events for hours; both billing and analytics have to credit those events to the period they happened in.
Worked solution and explanation
The trap
This is an exactly-once ledger hiding inside a telemetry firehose. Billing and analytics read the same device events with opposite correctness budgets. Billing needs every impression counted once, starts paired with completes, and a total that matches the ad server. Analytics can tolerate late, approximate counts. Anyone can draw Kafka into a warehouse. What separates candidates is refusing to let one streaming aggregator serve both. Count raw events and the first daily reconciliation misses tolerance: retries inflate the total, partial credit never gets applied, and the partner blocks payment.
Walk the requirements
Step 1: Dedup on impression_id before anything counts
Key a Flink job on impression_id and keep seen ids in state for the lateness window. A device retry with a known id is dropped before it ever reaches pairing. Counting raw events bills every retry twice.
Step 2: Pair start and complete in keyed state
Hold the start until its complete arrives. A complete inside the threshold emits full credit. A start whose timer fires past the threshold emits half credit. The rule lives in one place in code, so every impression gets the same treatment.
Step 3: Bucket by event time, not arrival time
Buffered devices deliver hours late. Each path owns a stage that assigns events to the period they happened in: a bucketing step after pairing on the billing side, and the daily rollup on the analytics side. A late event inside the agreed window lands in its own day, and the next reconciliation sees it. Arrival-time bucketing inflates today and starves the day the event actually happened.
Step 4: Gate the billing publish on reconciliation
Before the daily window, diff the billing fact against the ad server's record. Inside tolerance, publish. Outside, hold the day back. The object-storage archive of raw events is what you replay to explain a gap or rebuild a day.
The shape that fits
- device_telemetry (source, Kafka)
- impression_dedup (transform, Flink); freshness: real-time
- partial_credit_pairing (transform, Flink)
- event_time_billing_periods (transform, Flink)
- billing_warehouse (storage, warehouse)
- reconciliation_gate (quality gate, custom)
- ad_billing (consumer, API)
- event_archive (storage, object storage)
- event_time_daily_rollup (transform, Spark); freshness: < 24h
- analytics_warehouse (storage, warehouse)
- content_analytics_dashboard (consumer, dashboard)
- device_telemetry -> impression_dedup
- impression_dedup -> partial_credit_pairing
- partial_credit_pairing -> event_time_billing_periods
- event_time_billing_periods -> billing_warehouse
- billing_warehouse -> reconciliation_gate
- reconciliation_gate -> ad_billing
- device_telemetry -> event_archive
- event_archive -> event_time_daily_rollup
- event_time_daily_rollup -> analytics_warehouse
- analytics_warehouse -> content_analytics_dashboard
- Source + archive: device_telemetry, event_archive
- Billing path: impression_dedup, partial_credit_pairing, event_time_billing_periods, billing_warehouse, reconciliation_gate
- Analytics path: event_time_daily_rollup, analytics_warehouse
- Consumers: ad_billing, content_analytics_dashboard
- One stream for both
- A running count of raw events serves billing and analytics. Retries double-count, partial credit is never applied, and late events land on arrival day. Reconciliation fails on day one.
- Two paths, two budgets
- Billing dedups and pairs per
impression_id, then passes a reconciliation gate. Analytics batches from the archive. Each path pays only for the correctness it needs.
Common follow-up questions
- A complete arrives just past the qualifying threshold. What does billing show? (The timer already emitted half credit; a late complete must not promote it.)
- Reconciliation flags a gap with the ad server. How do you triage it? (Per-period diff of missing and extra ids, replayed from the raw archive.)
- A device buffers past the lateness window. Where does that event go? (Bounded reopen windows versus an explicit adjustment record.)