Seconds to Trend

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

mid

Asked in

TikTok

Problem

Design the pipeline that turns roughly 5 billion engagement events a day (views, likes, watch-time pings) from a short-video app into 2 outputs: a trending surface of the hottest videos and the daily active user and 7-day retention reports. Both read the same events, and streaming everything is not an option.

Business requirements

• When a video starts spiking, the trending team wants it on the surface within seconds, not on tomorrow's report.
• Growth reports daily active users and 7-day retention once a day and needs the counts to be exact.
• Finance flagged that streaming the full firehose for every consumer is expensive; only the consumers that need seconds should pay for it.

Worked solution and explanation

2 budgets, one event stream

This is a latency-budget split dressed up as a Kafka-and-Spark trivia question. Anyone can name the tools. What separates candidates is routing trending and growth down separate paths off the same durable log, and saying why out loud. Pick one path for both and you lose either way: stream everything and finance pays firehose compute for a once-a-day dashboard while DAU still drifts on late mobile events, or batch everything and the hot list arrives a day after the spike.

Step 1: Land every event in one replayable log

5 billion events a day with bursty peaks means the apps cannot push straight into consumers. Kafka partitioned by video_id absorbs the spike and gives both paths the same source of truth. Without it, a slow trending job backpressures the apps and a batch backfill has nothing to re-read.

Step 2: Put only trending on the streaming path

A Flink job keyed by video_id keeps a 5-minute rolling velocity and writes the hot list to Redis, which the trending API reads directly. Approximate is fine here, so at-least-once delivery is enough: one duplicate barely moves a ranking, and you skip the cost of exactly-once.

Step 3: Put DAU and retention on a daily batch

Growth publishes these numbers, so they must be exact and stable. A daily Spark job reads the date partition, dedups on user_id per day, and loads the warehouse. Reusing streaming counts would publish a figure that wobbles as late events land.

Step 4: Give late events a different answer on each path

Phones buffer offline and replay minutes late. The stream uses event-time watermarks and accepts that a late event may miss its trending window. The batch runs after the day closes, so a late event that lands before the run is still counted. Same data, 2 correctness budgets.

One shared streaming job

A single streaming job feeds a metrics table that everything reads. DAU drifts with late events, the bill covers 5B events a day for a daily report, and trending competes with heavy aggregation for cluster headroom.

Split by consumer

The stream carries only the windowed velocity state for trending. The full-history aggregation runs once a day on cheaper batch compute and publishes numbers that never move after release.

Common follow-up questions

• A creator complains their video trended for a second and vanished. How do you stabilize the signal without making it stale? (Tests window choices: longer or overlapping windows, decay, and minimum-volume thresholds while staying fresh within seconds.)
• Volume jumps 5x for a global event in 48 hours. What absorbs it and what breaks first? (Tests capacity thinking: Kafka partitions scale out, but windowed state and Redis write throughput are the first pressure points.)