The Next Track

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
medium
Seniority
mid
Asked in
Handshake

Problem

Every play, skip and save from our 500 million listeners arrives as an event, and our recommender has to reflect a listener's latest plays within a minute or two. Those same events feed a weekly model retraining over months of history. Design the pipeline that serves fresh session features to the recommender and lands training data for the weekly job.

Business requirements

Worked solution and explanation

Why this problem exists in real interviews

This is a lambda-style split wearing a recommendation-system costume. The real skill is routing one event stream into two paths: a fast path that keeps a listener's session state fresh for serving, and a slow path that lands exactly-once training history. Each guarantee has to live only on the path that needs it. The trap is treating 'recommendations' as one system with one clock. Stream everything and you pay streaming prices for a six-month genre affinity the weekly model learns anyway. Batch everything and a listener skips a track while the recommender keeps suggesting more of it for hours.

The whiteboard answer is a single stream writing to one table that the recommender queries, retrained off that same table. It looks clean until a few things break at once. The serving table bloats with history it never reads. The training job double-counts every event a phone retried. The offline metrics look great while the live model quietly disagrees with them.


Walk the requirements

Step 1: Buffer once, fan out to two speeds

Billions of play and skip events land in a durable buffer that two consumers read independently: the streaming feature job and the training landing. One ingestion, two speeds. Two separate ingest pipelines kept in sync is the version that drifts within a month. One buffer feeding both is what keeps the fast and slow paths reading identical events.

Step 2: Compute session features in the stream, serve from an online store

The streaming processor keeps per-listener session state (recent plays, skip rate, current context) and writes it to a low-latency online store, a feature store or key-value store. The recommender reads that store at request time, so a skip shows up in the next suggestion within a minute or two. This path is allowed to be approximate: a duplicate skip nudges a counter, it does not corrupt a ledger.

Step 3: Land training data exactly-once for the weekly retrain

The training path lands every event in a lake as an upsert keyed on event_id. A phone's retry or a stream replay overwrites the row it already wrote instead of adding a second play. Weekly retraining reads months of this history straight from the lake, and long-horizon signals are learned there. Exactly-once lives here and only here, because these are the labels the offline metrics are computed against.


The shape that fits

One event buffer feeds two clocks: streaming session features into an online store the recommender reads, and exactly-once history into a lake for the weekly retrain.
  • play_events (source, Kafka)
  • stream_features (transform, Flink); freshness: < 1min
  • online_feature_store (storage, Feast)
  • recommender (consumer, API)
  • training_lake (storage, Delta Lake); idempotency: upsert
  • model_training (transform, Spark)
  • play_events -> stream_features
  • play_events -> training_lake
  • stream_features -> online_feature_store
  • online_feature_store -> recommender
  • training_lake -> model_training
Stream everything
The streaming job computes six-month genre affinity and embeddings that only move weekly. It pays real-time compute for batch-cadence signals and bloats the online store. Cost climbs with no freshness benefit.
Split by change rate
Only session-scoped state runs in the stream, and long-horizon signals are learned by the weekly retrain from the lake. The hot path stays small and cheap, and freshness lands where it matters.

Common follow-up questions