Live Viewers, Live Billing

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L4

Problem

Advertisers keep disputing our impression invoices, and creators complain their live viewer count trails the chat. A top stream draws 200k concurrent viewers, and an ad break fires every viewer's impression within the same second. Design the pipeline for our livestream events.

The stream is live. The data cannot wait.

Business requirements

• Creators watch their concurrent viewer count and chat rate during a stream and need it to feel live, not a few minutes behind.
• Advertisers pay only for ads a viewer actually watched; an under-count loses revenue and an over-count creates contractual risk.

Worked solution and explanation

Two readers, two correctness budgets

This is one event stream split for two readers whose budgets pull in opposite directions, dressed up as livestream analytics. Creators want a viewer count that feels live and can be a little off. Advertisers pay only for ads a viewer watched, counted once. Anyone draws a Kafka topic and a stream processor. The trap is the single aggregator both read: sized to feel live, it double-counts on player retries; sized to be exact, it lags behind chat. Either way the next invoice gets disputed.

The default reach is one streaming job that writes a shared metrics table that the creator dashboard and ads billing both query. Creators feel the lag of the heavier billing logic running beside their counts. Billing inherits the dashboard's tolerance for approximation, so a retried impression counts twice and a requested ad that never played gets invoiced. One table, two unhappy readers.

Walk the requirements

Step 1: Stream creator metrics live and accept approximation

creator_metrics_stream runs on Flink at real-time freshness and writes rolling viewer and chat counts into creator_metrics_store, a Redis serving layer that creator_dashboard polls every few seconds. Approximate is fine here: a viewer mid-reconnect counted twice for one heartbeat costs nothing. Holding counts until they are exact is the version where creators notice the lag on air.

Step 2: Bill each watched ad once, on a stable id

Every impression carries a stable impression id from the player. dedup_watched_impressions drops repeats of that id, keeps only ads the viewer actually watched, and writes to billable_impression_store, which ads_billing reads. Mobile players retry on flaky networks, so without the dedup one ad break bills several times. Without the watched check, an ad that never played gets invoiced. Flink's checkpointed state lets the dedup survive a restart without dropping or repeating an id.

The shape that fits

Only creator_metrics_stream carries a freshness chip. The billing side runs on Flink for its exactly-once state, not for speed: finance reads one report a day, and the store it reads is durable and queryable rather than the Redis layer the creator view hits.

Common follow-up questions

• An impression is logged by the player, but the watch-completion signal never arrives. What does the ads path do, and how does it eventually decide? (Tests whether the candidate holds pending impressions in state for a bounded window and either bills them when the watch signal lands or expires them unbilled, so the billable count converges.)
• A creator's dashboard freezes during a viral stream because the creator aggregator is overloaded. What in this design keeps ads billing working? (Tests whether the candidate sees the two paths as independent consumers of the same topic, each with its own state and throughput, so overload on one does not propagate.)
• Phones buffer events and send them up to ten minutes late. Which path cares, and what changes? (Tests whether the candidate lets the creator view ignore stragglers while the billing path waits long enough to count a late watched ad once.)