Seconds to Trend

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
medium
Seniority
mid
Asked in
TikTok

Problem

We run a short-video platform where roughly 5 billion engagement events a day (views, likes, watch-time pings) come off the apps. The trending team needs the hottest videos surfaced within seconds of a spike, while the growth team reports daily active users and 7-day retention on a T+1 cadence from the same events. Design the pipeline that serves both consumers without paying to stream everything.

Business requirements

Worked solution and explanation

Why this problem exists in real interviews

This looks like a Kafka-and-Spark trivia question but it is really a two-speed problem hiding behind one event stream. Trending wants engagement velocity within seconds; growth wants exact DAU and retention once a day. The trap is picking a single answer for both: stream everything and you pay firehose compute to serve a daily report, or batch everything and trending is always one day stale. The skill being probed is whether you split consumers by their actual latency budget and justify the split out loud.

The default whiteboard reach is one streaming job that writes a metrics table everything reads. It feels modern, but DAU computed on a streaming aggregate is approximate (late mobile events drift the count), reporting publishes a number that wobbles, and finance gets a bill for streaming 5 billion events a day to feed a once-a-day dashboard. Meanwhile trending shares cluster headroom with the heavy daily aggregation and lags exactly when a video spikes.

Walk the requirements

Step 1: Buffer the firehose in one log both paths read

5 billion events a day with 4x peak bursts means producers cannot push straight into consumers. Kafka partitioned by video_id absorbs the spike and gives both the stream tier and the batch tier the same replayable source. Without this buffer, a trending slowdown backpressures the apps and a batch backfill has nowhere to re-read from.

Step 2: Put only trending on the streaming path

Trending is engagement velocity over a rolling 5-minute window, needed within seconds. A stream processor (Flink or Spark Structured Streaming) keys by video_id, maintains the windowed count, and writes the hot list to a serving store the trending surface queries directly. At-least-once is acceptable here because one duplicate barely moves a velocity ranking, so you avoid the cost of exactly-once on the fast path.

Step 3: Put DAU and retention on the batch path

DAU and 7-day retention are T+1 and feed external reporting, so they must be exact and stable once published. A daily Spark job reads the day-partitioned log, dedups on (user_id, date), and lands aggregates in the warehouse. Reusing the streaming approximate counts here would publish a number that drifts as late events arrive. This is the consumer that should NOT be streamed.

Step 4: Decide what each path does with late mobile events

Clients buffer offline and replay events minutes late. The stream tier uses event-time watermarks and accepts that a late event may miss its trending window, which is fine for an approximate velocity metric. The batch tier sidesteps it entirely: it runs after the day closes over the date partition, so a late event that lands before the run is still counted exactly. Same data, two correctness budgets.

The shape that fits

One Kafka log feeds two paths sized to two budgets: seconds-and-approximate for trending, T+1-and-exact for growth reporting.
  • engagement_events (source, Kafka, 5B/day, partitioned by video_id)
  • trending_stream (transform, Flink, 5-min rolling velocity); freshness: < 1min
  • trending_store (storage, Redis); freshness: < 1min
  • trending_surface (consumer, Trending API); freshness: < 1min
  • daily_aggregation (transform, Spark, date-partitioned, dedup on user/date); freshness: < 24h
  • analytics_warehouse (storage, BigQuery); freshness: < 24h
  • growth_reporting (consumer, Looker); freshness: < 24h
  • engagement_events -> trending_stream
  • trending_stream -> trending_store
  • trending_store -> trending_surface
  • engagement_events -> daily_aggregation
  • daily_aggregation -> analytics_warehouse
  • analytics_warehouse -> growth_reporting
  • Ingestion: engagement_events
  • Streaming path: trending_stream, trending_store, trending_surface
  • Batch path: daily_aggregation, analytics_warehouse, growth_reporting

Common follow-up questions