Where the Crowd Goes

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
medium
Seniority
mid
Asked in
Whatnot

Problem

What pipeline would you put behind the API that serves our live-shopping home screen's leaderboard of the livestreams hottest right now, scored on viewer joins, bids and reactions? Engagement peaks near 500,000 events per second during big drops, and millions of open apps poll the board every few seconds.

Business requirements

Worked solution and explanation

Why this problem exists in real interviews

The ask reads like 'build the pipeline behind a leaderboard API', and that framing is the trap. Candidates spend the hour on request shapes, auth and pagination, and never notice that the whole difficulty lives BEHIND the endpoint. This is a freshness-versus-read-scale problem wearing an API costume: millions of clients poll a board that must move within seconds of a live spike. The moment you let the API compute rankings, you have coupled read QPS to compute cost and you lose both. The real question is whether you split the system into a streaming path that decides WHAT the ranking is and a serving path that decides how fast you can HAND IT OUT.

The naive design has the API query the event store and rank on the fly. It demos fine with one viewer. At a few million pollers every few seconds, the ranking query fans out over the raw event volume on every request, p99 latency detonates, and the same database that computes rankings is now hammered by reads, so freshness dies exactly when engagement is highest. You cannot cache your way out either, because a cache short enough to feel live is a cache that never hits.


Break down the requirements

Step 1: Buffer the engagement firehose

Joins, bids and reactions peak around half a million per second during a drop. Land them in a durable log first. That gives the stream processor a shock absorber for bursts and, just as important, a replay log: when the processor restarts mid-drop, it resumes from its offset instead of dropping the exact window of events that mattered most.

Step 2: Score on a sliding window, not a running total

'Hot right now' is engagement VELOCITY over a recent window (say 1 to 5 minutes), not lifetime popularity, or the same big sellers sit on top forever. A sliding window lets a stream that was quiet a minute ago climb within seconds of spiking; a tumbling window would make the board jump at boundaries. Rankings are an approximate ordering, so at-least-once with idempotent aggregation is enough. Do not spend an exactly-once budget reshuffling a board every few seconds.

Step 3: Publish scores to a read-optimized serving store

The stream processor writes the current top-N into a low-latency serving store, and the API's job shrinks to a keyed lookup that returns in single-digit milliseconds. This is the load-bearing decision: the store decouples read scale from compute, so the endpoint holds its latency under millions of pollers while the streaming job independently keeps the numbers fresh.

Step 4: Make staleness loud

The dangerous failure is a frozen board still returning 200s. Watch how old the scores in the serving store are and page when they fall behind. Because reads come from the store, an outage in the compute path degrades to a stale-but-available board, which beats an outage only if you actually measure the staleness and someone hears about it.


The shape that fits

Engagement events land in a replayable Kafka log, a streaming job scores a sliding window and writes top-N into a serving store, the API reads only that store, and a monitor pages when the stored scores go stale.
  • engagement_events (source, Kafka)
  • ranking_stream (transform, Flink)
  • rankings_store (storage, Redis); freshness: real-time
  • rankings_api (consumer, API)
  • freshness_monitor (consumer, Prometheus); alerting: page if board lag > 1min
  • engagement_events -> ranking_stream
  • ranking_stream -> rankings_store
  • rankings_store -> rankings_api
  • rankings_store -> freshness_monitor

Common follow-up questions