Where the Crowd Goes
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- mid
- Asked in
- Whatnot
Problem
We run a live-shopping marketplace where the home screen shows a leaderboard of the hottest livestreams right now, scored by live engagement like viewers joining, bids landing, and reactions. Design the pipeline behind the API that serves this leaderboard: millions of clients poll it, and a stream that was quiet a minute ago can suddenly spike, so the board has to reflect engagement within seconds while read traffic never stalls the ranking updates.
Business requirements
- When a stream starts popping off, it should climb the board almost immediately, not on the next hourly refresh.
- The whole app hits this endpoint constantly and read traffic keeps growing; it can't get slower or knock out the ranking updates when everyone opens the app at once.
- During a big drop we get sudden floods of activity, and if a processor restarts we can't just lose those events.
- The worst case is the board looks fine but is frozen ten minutes behind reality and nobody notices.
Worked solution and explanation
Why this problem exists in real interviews
The prompt says 'design a REST API for rankings' and that framing is the trap. Candidates spend the hour on request/response shapes, auth, and pagination, and never notice that the whole difficulty lives BEHIND the endpoint. This is a freshness-versus-read-scale problem wearing an API costume: millions of clients poll a board that must move within seconds of a live spike. The moment you decide the API computes rankings, you have coupled read QPS to compute cost and you lose both. The real question is whether you split the system into a streaming path that decides WHAT the ranking is and a serving path that decides how fast you can HAND IT OUT.
The naive design has the API query the event store and rank on the fly. It demos fine with one viewer. At a few million pollers every few seconds, the ranking query fans out over the raw event volume on every request, p99 latency detonates, and the same database that computes rankings is now getting hammered by reads, so freshness dies exactly when engagement is highest. You cannot cache your way out either, because a cache short enough to feel live is a cache that never hits.
Break down the requirements
Step 1: Buffer the engagement firehose
Joins, bids, and reactions arrive in the hundreds of thousands per second and spike hard during a drop. Land them in a durable queue first. That gives the stream processor a shock absorber for bursts and, just as important, a replay log: when the processor restarts mid-drop, it resumes from its offset instead of dropping the exact window of events that mattered most.
Step 2: Score on a sliding window, not a running total
'Hot right now' is engagement VELOCITY over a recent window (say 1 to 5 minutes), not lifetime popularity, or the same big sellers sit on top forever. Use a sliding window so a stream that was quiet a minute ago climbs within seconds of spiking. A tumbling window would make the board jump at boundaries; sliding keeps it smooth. Rankings are an approximate ordering, so at-least-once with idempotent aggregation is enough; do not spend an exactly-once budget reshuffling a board every few seconds.
Step 3: Publish scores to a read-optimized serving store
The stream processor writes the current top-N (globally and per category/region) into a low-latency serving store. The API's job shrinks to a keyed lookup that returns in single-digit milliseconds. This is the load-bearing decision: the store decouples read scale from compute, so the endpoint holds its latency budget under millions of pollers while the streaming job independently keeps the numbers fresh.
Step 4: Make staleness loud, and degrade gracefully
The dangerous failure is a frozen board still returning 200s. Track event-processing lag and ranking freshness; alert when the window falls behind. Because reads come from the serving store, an outage in the compute path degrades to a stale-but-available board carrying a freshness timestamp, which beats an outage as long as you actually measure and surface the staleness.
The shape that fits
- engagement_events (source, Kafka); parallelism: high-partition
- ranking_stream (transform, Flink); freshness: real-time
- rankings_store (storage, Redis); freshness: < 1min
- rankings_api (consumer, REST service); freshness: < 1min
- freshness_monitor (consumer, Prometheus + Grafana); alerting: Ranking window lag above threshold
- engagement_events -> ranking_stream
- ranking_stream -> rankings_store
- rankings_store -> rankings_api
- ranking_stream -> freshness_monitor
- rankings_store -> freshness_monitor
- Ingest: engagement_events
- Compute: ranking_stream
- Serve + observe: rankings_store, rankings_api, freshness_monitor
Common follow-up questions
- A single mega-stream draws 20x the viewers of anything else and its partition becomes a hotspot that lags the whole job. How do you keep it from stalling everyone else's rankings? (Tests key-skew remediation: sub-keying or salting the hot stream, isolating its aggregation, or pre-aggregating hot keys so one viral room doesn't back up the window for tens of thousands of others.)
- Product now wants rankings personalized per viewer instead of one global board. What in this design changes, and what breaks? (Tests whether the candidate sees that a single precomputed top-N no longer serves everyone: they need per-segment precomputation plus a feature/serving layer, and must reason about the explosion in serving-store cardinality versus computing a personalized re-rank at read time.)