The Same Stream Twice

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

senior

Asked in

The Walt Disney Company

Problem

What pipeline carries about 2 billion playback heartbeats a day from 150 million subscribers to two readers at once: an on-call page when a title starts rebuffering in a region, and a daily studio royalty report on exact minutes-watched?

Business requirements

• When a title starts rebuffering in a region, I need on-call paged within seconds, not in tomorrow's report.
• We pay studios on exact minutes-watched, so the daily total cannot double-count a retried event or drop one that slipped a window.
• A buggy device firmware release once sent malformed heartbeats and took our whole pipeline down; that can never block live alerting again.
• I don't want two separate collection paths drifting apart; both teams should be reading the same source of truth, each on their own offsets.

Worked solution and explanation

Why this problem exists in real interviews

This is two consumers with opposite correctness budgets reading one event stream, disguised as a streaming dashboard. Reliability wants approximate-and-instant: a rebuffering spike per title and region in seconds, off-by-a-few is fine. Finance wants exact-and-patient: royalties on minutes-watched, where one double-counted retry is a contract dispute. The trap is the single aggregation serving both: too rough to bill on, and once you bolt exactness onto it, too slow to page on-call.

The whiteboard answer is one stream writing a minutes_watched table both the QoE dashboard and the royalty report read. It looks clean for a week. Then a producer retries during a deploy, both counts inflate, and finance overpays a studio. Or a malformed heartbeat crashes the consumer, it replays the same offset, and reliability goes blind for hours. Three requirements failed quietly because they were never separated.

Walk the requirements

Step 1: Land everything in one durable queue

Heartbeats land in a partitioned message queue first. The stream processor and the daily batch each read from their own committed offsets, so neither starves the other and either can replay alone. The subtle failure is chaining the royalty batch off the stream processor's output instead of the raw queue: royalties then inherit the live path's approximations and lose independent replay. Two separate collection paths drift just as badly, and the day they disagree nobody can say which number is right.

Step 2: Stream the QoE path; approximate is the budget

A stream processor windows heartbeats by title and region, flags rebuffering above a baseline, and emits to an alert destination inside < 1min. Counting a viewer mid-transition slightly wrong is fine; live-feel is the budget. The 'compute it exactly before we alert' version is where on-call learns of the incident from social media.

Step 3: Make royalty exactly-once at the sink, not in the stream

The daily batch reads raw heartbeats straight from the queue, where each event carries a stable event_id from the player. It dedups on that id and upserts into the warehouse, so a producer retry is absorbed and a window-edge event is not dropped. Exactly-once is enforced at the sink with a key, not hoped for in the streaming layer. Billing from the approximate live table is the mistake that becomes an overpayment.

Step 4: Quarantine poison pills, keep draining

Validation lives on the consumer: a heartbeat that fails schema (the missing title_id from the bad firmware build) routes to a dead-letter quarantine and its offset is acked, so the main path keeps moving. A separate process triages the quarantine, and its depth is alerted on so a firmware regression is visible. If a bad event can halt the consumer, one device build blinds reliability during the incident it most needed to see.

The shape that fits

Common follow-up questions

• A heartbeat arrives two hours after its day already closed in the royalty warehouse. What does the batch do with it? (Tests bounded-lateness handling plus an `event_id`-keyed correction run that lands the late event in the next cycle without double-counting.)
• A tentpole release pushes the live path to 5x volume and consumer lag climbs. What keeps on-call from going blind? (Tests partitioning, autoscaling on lag, and the decoupling that stops the royalty batch from stealing the live path's resources.)