The Thirty-Second Rule

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
medium
Seniority
mid
Asked in
Handshake

Problem

A music-streaming platform emits a play event every time someone starts, pauses, or skips a track, and three teams read that same feed: recommendations wants what a listener is playing reflected in their home feed within a minute, royalty accounting must count each qualifying play (30 seconds or more of listening) exactly once because payouts to rights-holders can't be clawed back, and product analytics reports daily engagement the next morning. Design the platform that ingests the play stream once and serves all three from it at their own freshness and correctness needs.

Business requirements

Worked solution and explanation

Why this problem exists in real interviews

This is a multi-consumer fan-out problem wearing a music-app costume. One play-event feed has to satisfy three teams with three different correctness budgets: recommendations wants freshness and shrugs at a slightly wrong count, royalty accounting needs each qualifying play counted exactly once because a payout to a rights-holder cannot be clawed back, and analytics just wants yesterday's numbers by morning. The trap is a single pipeline that averages all three: it is too slow to feel live, too approximate to bill on, and too expensive because it streams work that could have waited until 2am.

The default whiteboard answer is one Kafka topic into one stream job that writes a 'plays' table everyone queries. Recommendations lag because the job is doing heavy royalty logic inline. Royalties double-count the first time a producer retries a batch of offline uploads, because nothing dedups on a stable id. Analytics runs full scans over a table that was never partitioned for it. Three consumers, three quiet failures, one shared pipeline to blame.

Walk the requirements

Step 1: Ingest the play stream once, into a shared queue

Every start, pause, skip, and progress event lands in one high-throughput queue partitioned so a hot artist or a peak-hour region does not create a single hot partition. All three consumers read from here as independent consumer groups, so one team's backpressure never stalls another's. Three separate ingest layers would triple cost and let the three views of a play drift apart.

Step 2: Stream recommendations to a serving store; approximate is fine

A streaming processor keeps per-listener recent activity and pushes it to a low-latency serving store the home feed reads within a minute. If a viewer's count is briefly off by one during a skip, nobody notices. The mistake here is doing the royalty dedup inline on this path: it is the heavy work that makes the feed lag.

Step 3: Count royalties exactly once on a stable play id

A qualifying play is a start event whose progress signal shows 30+ seconds, correlated on the play id the client stamped. The royalty path dedups on that id and upserts counts into a warehouse idempotently, so a producer retry or a stream restart replays without inflating what a rights-holder gets paid. Latency is not the budget; a duplicated cent is, once it scales to millions of plays.

Step 4: Batch analytics on its own cadence and layout

Daily engagement lands the next morning from a batch job that reads the same events but writes a warehouse table partitioned by date and the dimensions analysts filter on. Streaming this path adds cost and operational surface for a consumer that only ever looks at yesterday.

The shape that fits

One play stream, three consumer groups: streaming-and-approximate for recommendations, exactly-once for royalties, nightly batch for analytics.
  • play_events (source, Kafka)
  • reco_stream (transform, Flink)
  • reco_serving (storage, Redis)
  • home_feed (consumer, API)
  • royalty_aggregator (transform, Flink)
  • royalty_warehouse (storage, Snowflake)
  • analytics_batch (transform, Spark)
  • analytics_warehouse (storage, Snowflake)
  • play_events -> reco_stream
  • play_events -> royalty_aggregator
  • play_events -> analytics_batch
  • reco_stream -> reco_serving
  • reco_serving -> home_feed
  • royalty_aggregator -> royalty_warehouse
  • analytics_batch -> analytics_warehouse
  • Ingest: play_events
  • Three paths: reco_stream, royalty_aggregator, analytics_batch
  • Serving + warehouses: reco_serving, home_feed, royalty_warehouse, analytics_warehouse

Common follow-up questions