Where the Crowd Goes
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- mid
- Asked in
- Whatnot
Problem
What pipeline would you put behind the API that serves our live-shopping home screen's leaderboard of the livestreams hottest right now, scored on viewer joins, bids and reactions? Engagement peaks near 500,000 events per second during big drops, and millions of open apps poll the board every few seconds.
Business requirements
- When a stream starts popping off, it should climb the board almost immediately, not on the next hourly refresh.
- The whole app hits this endpoint constantly and read traffic keeps growing; it can't get slower or knock out the ranking updates when everyone opens the app at once.
- During a big drop we get sudden floods of activity, and if a processor restarts we can't just lose those events.
- The worst case is the board looks fine but is frozen ten minutes behind reality and nobody notices.
Worked solution and explanation
Why this problem exists in real interviews
The ask reads like 'build the pipeline behind a leaderboard API', and that framing is the trap. Candidates spend the hour on request shapes, auth and pagination, and never notice that the whole difficulty lives BEHIND the endpoint. This is a freshness-versus-read-scale problem wearing an API costume: millions of clients poll a board that must move within seconds of a live spike. The moment you let the API compute rankings, you have coupled read QPS to compute cost and you lose both. The real question is whether you split the system into a streaming path that decides WHAT the ranking is and a serving path that decides how fast you can HAND IT OUT.
The naive design has the API query the event store and rank on the fly. It demos fine with one viewer. At a few million pollers every few seconds, the ranking query fans out over the raw event volume on every request, p99 latency detonates, and the same database that computes rankings is now hammered by reads, so freshness dies exactly when engagement is highest. You cannot cache your way out either, because a cache short enough to feel live is a cache that never hits.
Break down the requirements
Step 1: Buffer the engagement firehose
Joins, bids and reactions peak around half a million per second during a drop. Land them in a durable log first. That gives the stream processor a shock absorber for bursts and, just as important, a replay log: when the processor restarts mid-drop, it resumes from its offset instead of dropping the exact window of events that mattered most.
Step 2: Score on a sliding window, not a running total
'Hot right now' is engagement VELOCITY over a recent window (say 1 to 5 minutes), not lifetime popularity, or the same big sellers sit on top forever. A sliding window lets a stream that was quiet a minute ago climb within seconds of spiking; a tumbling window would make the board jump at boundaries. Rankings are an approximate ordering, so at-least-once with idempotent aggregation is enough. Do not spend an exactly-once budget reshuffling a board every few seconds.
Step 3: Publish scores to a read-optimized serving store
The stream processor writes the current top-N into a low-latency serving store, and the API's job shrinks to a keyed lookup that returns in single-digit milliseconds. This is the load-bearing decision: the store decouples read scale from compute, so the endpoint holds its latency under millions of pollers while the streaming job independently keeps the numbers fresh.
Step 4: Make staleness loud
The dangerous failure is a frozen board still returning 200s. Watch how old the scores in the serving store are and page when they fall behind. Because reads come from the store, an outage in the compute path degrades to a stale-but-available board, which beats an outage only if you actually measure the staleness and someone hears about it.
The shape that fits
- engagement_events (source, Kafka)
- ranking_stream (transform, Flink)
- rankings_store (storage, Redis); freshness: real-time
- rankings_api (consumer, API)
- freshness_monitor (consumer, Prometheus); alerting: page if board lag > 1min
- engagement_events -> ranking_stream
- ranking_stream -> rankings_store
- rankings_store -> rankings_api
- rankings_store -> freshness_monitor
Common follow-up questions
- A single mega-stream draws 20x the viewers of anything else and its partition becomes a hotspot that lags the whole job. How do you keep it from stalling everyone else's rankings? (Tests key-skew remediation: salting the hot stream's key, isolating its aggregation, or pre-aggregating hot keys so one viral room does not back up the window for tens of thousands of others.)
- Product now wants rankings personalized per viewer instead of one global board. What in this design changes, and what breaks? (Tests whether the candidate sees that one precomputed top-N no longer serves everyone: per-segment precomputation versus a re-rank at read time, and the explosion in serving-store cardinality.)