The Next Track
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- mid
- Asked in
- Handshake
Problem
We run a music-streaming service where every play, skip, and save from 500 million listeners arrives as an event, and the recommender needs a listener's just-played tracks reflected in their suggestions within a minute or two. Those same events also feed the weekly model retraining that runs over months of history. Design the pipeline that serves fresh session features to the recommender in near real time while landing exactly-once training data for the batch jobs.
Business requirements
- When a listener skips a track, the very next recommendation should already know about it.
- The weekly model has to train on every event once, with no dropped or double-counted plays skewing the labels.
- We do not want two separate event pipelines to keep in sync; the same events should power both fast serving and slow training.
Worked solution and explanation
Why this problem exists in real interviews
This is a lambda-style split wearing a recommendation-system costume. The real skill: can you route one event stream into a fast path that keeps a listener's session features fresh for serving and a slow path that lands exactly-once training data, and can you say WHICH features belong on which path. The trap is treating 'recommendations' as one system with one clock. Stream everything and you pay streaming prices to compute a 30-day genre affinity that changes once a week; batch everything and a listener skips a track while the recommender keeps suggesting more of it for hours.
The whiteboard answer is a single stream that writes to a table the recommender queries, retrained off the same table nightly. It looks clean until the online store is carrying features that only ever move at batch cadence, the training job double-counts every event a producer retried, and the offline metrics look great while the live model quietly disagrees with them because serving and training compute features differently.
Walk the requirements
Step 1: Buffer once, fan out to two speeds
Billions of play and skip events land in a durable queue that both consumers read independently: the streaming feature job and the batch training job. One ingestion, two speeds. Building two separate ingest pipelines to keep in sync is the version that drifts within a month; the same buffer feeding both is what keeps the fast and slow paths reading identical events.
Step 2: Compute session features in the stream, serve from an online store
The streaming processor maintains per-listener session state (recent plays, skip rate, current context) and writes it to a low-latency online store (a feature store or key-value store). The recommender reads that store at request time, so a skip is reflected in the next suggestion within a minute or two. This path is allowed to be approximate: a duplicate skip nudges a counter, it does not corrupt a ledger.
Step 3: Land training data exactly-once for the weekly retrain
The batch path writes events into a lake, keyed on event_id with idempotent partition-overwrite, so a producer retry or a stream replay does not double-count plays. Weekly retraining reads months of this history. Exactly-once lives here and only here, because these are the labels the offline metrics are computed against; paying for it on the serving path would just make the hot path slower for no correctness gain.
The shape that fits
- play_events (source, Kafka); parallelism: high-cardinality by listener_id
- stream_features (transform, Flink); freshness: real-time
- online_feature_store (storage, Feast (Redis-backed)); freshness: < 1min
- recommender (consumer, Serving API); freshness: < 1min
- training_lake (storage, Delta Lake); backfill: partition_overwrite; idempotency: upsert
- batch_features (transform, Spark); freshness: < 24h
- model_training (transform, Spark)
- play_events -> stream_features
- play_events -> training_lake
- stream_features -> online_feature_store
- online_feature_store -> recommender
- batch_features -> online_feature_store
- training_lake -> batch_features
- training_lake -> model_training
- Ingest: play_events
- Fast path: stream_features, online_feature_store, recommender
- Slow path: training_lake, batch_features, model_training
- Stream everything
- The streaming job computes 30-day genre affinity and embeddings that only move weekly, paying real-time compute for batch-cadence features and bloating the online store. Cost climbs with no freshness benefit.
- Split by change rate
- Only session-scoped features run in the stream; long-horizon features are batch-computed and pushed into the online store slowly. The hot path stays small and cheap, freshness lands where it matters.
Common follow-up questions
- A regional evening peak pushes event volume to 10x for two hours. What backs up, and what stays within the one-to-two minute serving freshness target? (Tests backpressure thinking: the buffer absorbs the spike, the streaming job may lag but the online store still serves last-known features; the batch path is unaffected because it is not on the clock.)
- The team wants to add a new session feature and use it in training too. How do you get it into both paths without them drifting apart? (Tests training-serving consistency: a shared feature definition plus a point-in-time backfill over the lake so the offline values match what the stream would have produced.)