What Everyone Is Watching
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L6
Problem
We need to track what our subscribers are watching. This data feeds everything from our recommendation models to operations dashboards that monitor playback quality in real time. Design a data pipeline for our viewing events.
Someone is watching. Capture everything.
Business requirements
- Operations watches concurrent viewer counts during live events and outages; data science training tolerates T+1.
- Operations dashboards, recommendation models, the A/B platform, and content planning each have different access patterns; one storage layer can't serve all four at this scale.
- Most queries hit the last week, some go back ninety days, and queries past that are rare; storage cost dominates, so the full history belongs on cheap storage, and recent queries can't slow down by reading through old data.
- Playback quality events are how SRE detects outages; missing events translate to blind spots that delay incident response.
Worked solution and explanation
This is one event stream feeding four different access patterns, dressed up as "track what people watch". Anyone can land the events in a warehouse. The trap is letting one store serve ops, recommendations, experiments and planning, and letting the SRE signal ride best-effort delivery. Do that and live dashboards lag in the minutes they matter, the bill follows retention, and a dropped playback event becomes a blind spot during an outage.
- viewing_events (source, Kafka)
- event_archive (storage, Iceberg)
- ops_stream (transform, Flink); freshness: real-time
- ops_store (storage, PostgreSQL)
- playback_buffer (queue, Kafka)
- sre_stream (transform, Flink)
- recommendation_features (storage, PostgreSQL); freshness: < 24h
- hot_tier (storage, warehouse)
- archive_query (transform, SQL)
- ops_team (consumer, Grafana)
- sre (consumer, Grafana)
- recommendation_team (consumer, Jupyter)
- ab_team (consumer, Metabase)
- content_planning (consumer, Metabase)
- viewing_events -> event_archive
- viewing_events -> ops_stream
- viewing_events -> playback_buffer
- playback_buffer -> sre_stream
- ops_stream -> ops_store
- event_archive -> recommendation_features
- event_archive -> hot_tier
- event_archive -> archive_query
- ops_store -> ops_team
- sre_stream -> sre
- recommendation_features -> recommendation_team
- hot_tier -> ab_team
- hot_tier -> content_planning
- archive_query -> content_planning
Step 1: Split the cadences after ingest
ops_stream keeps concurrent viewer counts moving within seconds and writes to ops_store. Training tolerates T+1, so recommendation_features refreshes daily off the archive. One shared path would either starve ops or make training pay for streaming compute.
Step 2: Give each consumer its own store
Ops reads a small, fast ops_store. Recommendations read precomputed rows in recommendation_features. The A/B platform and content planning read the hot_tier warehouse. The paths diverge right after ingest, so a heavy planning query never slows the live dashboard.
Step 3: Tier by age, not by hope
event_archive is cheap lake storage holding everything, partitioned by date. Only recent data is copied into hot_tier. Anything older, say past ninety days, is reached through archive_query, which is slower but costs only what it scans. The bill follows access, not retention.
Step 4: Buffer the playback quality events
playback_buffer is a durable Kafka topic between ingest and sre_stream. If the consumer restarts, events wait in the log and are replayed, so SRE does not lose the moments just before an outage.
Common follow-up questions
- A live event drives 10x concurrent viewers. What does this design do, and what does ops see? (Tests whether you see that Kafka absorbs the spike and `ops_stream` scales independently, while batch paths on other stores are unaffected.)
- An audit needs data from over a year ago. What serves it? (Tests whether you point to `event_archive` through `archive_query` instead of hot storage.)