The Next Track
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- mid
- Asked in
- Handshake
Problem
Every play, skip and save from our 500 million listeners arrives as an event, and our recommender has to reflect a listener's latest plays within a minute or two. Those same events feed a weekly model retraining over months of history. Design the pipeline that serves fresh session features to the recommender and lands training data for the weekly job.
Business requirements
- When a listener skips a track, the very next recommendation should already know about it.
- The weekly model has to train on every event once, with no dropped or double-counted plays skewing the labels.
- We do not want two separate event pipelines to keep in sync; the same events should power both fast serving and slow training.
Worked solution and explanation
Why this problem exists in real interviews
This is a lambda-style split wearing a recommendation-system costume. The real skill is routing one event stream into two paths: a fast path that keeps a listener's session state fresh for serving, and a slow path that lands exactly-once training history. Each guarantee has to live only on the path that needs it. The trap is treating 'recommendations' as one system with one clock. Stream everything and you pay streaming prices for a six-month genre affinity the weekly model learns anyway. Batch everything and a listener skips a track while the recommender keeps suggesting more of it for hours.
The whiteboard answer is a single stream writing to one table that the recommender queries, retrained off that same table. It looks clean until a few things break at once. The serving table bloats with history it never reads. The training job double-counts every event a phone retried. The offline metrics look great while the live model quietly disagrees with them.
Walk the requirements
Step 1: Buffer once, fan out to two speeds
Billions of play and skip events land in a durable buffer that two consumers read independently: the streaming feature job and the training landing. One ingestion, two speeds. Two separate ingest pipelines kept in sync is the version that drifts within a month. One buffer feeding both is what keeps the fast and slow paths reading identical events.
Step 2: Compute session features in the stream, serve from an online store
The streaming processor keeps per-listener session state (recent plays, skip rate, current context) and writes it to a low-latency online store, a feature store or key-value store. The recommender reads that store at request time, so a skip shows up in the next suggestion within a minute or two. This path is allowed to be approximate: a duplicate skip nudges a counter, it does not corrupt a ledger.
Step 3: Land training data exactly-once for the weekly retrain
The training path lands every event in a lake as an upsert keyed on event_id. A phone's retry or a stream replay overwrites the row it already wrote instead of adding a second play. Weekly retraining reads months of this history straight from the lake, and long-horizon signals are learned there. Exactly-once lives here and only here, because these are the labels the offline metrics are computed against.
The shape that fits
- play_events (source, Kafka)
- stream_features (transform, Flink); freshness: < 1min
- online_feature_store (storage, Feast)
- recommender (consumer, API)
- training_lake (storage, Delta Lake); idempotency: upsert
- model_training (transform, Spark)
- play_events -> stream_features
- play_events -> training_lake
- stream_features -> online_feature_store
- online_feature_store -> recommender
- training_lake -> model_training
- Stream everything
- The streaming job computes six-month genre affinity and embeddings that only move weekly. It pays real-time compute for batch-cadence signals and bloats the online store. Cost climbs with no freshness benefit.
- Split by change rate
- Only session-scoped state runs in the stream, and long-horizon signals are learned by the weekly retrain from the lake. The hot path stays small and cheap, and freshness lands where it matters.
Common follow-up questions
- A regional evening peak pushes event volume to 10x for two hours. What backs up, and what stays within the one-to-two minute serving freshness target? (Tests backpressure thinking: the buffer absorbs the spike, and the streaming job may lag while the online store keeps serving last-known features. The training path is unaffected because it is not on the clock.)
- The team wants to add a new session feature and use it in training too. How do you get it into both paths without them drifting apart? (Tests training-serving consistency: a shared feature definition plus a point-in-time backfill over the lake, so the offline values match what the stream would have produced.)