Near-Real-Time Trending Dishes Dashboard

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L5

Problem

Design the data pipeline behind a dashboard that shows restaurant partners on our food delivery platform which dishes are trending in their city, judged on each dish's order rate over the last 30 minutes. The partner team plans for about 2 million orders a day across 300 cities, peaking near 500 orders per second at dinner.

The dish rankings update faster than the kitchen.

Business requirements

• Restaurant partners want to see what's trending in their city right now, not yesterday; a few minutes is fine.
• A dish counts as trending only when its current rate is well above its own historical pace; common dishes shouldn't trend just because they have steady volume.
• A viral social media post can drive a single dish to spike many times normal; that can't break the dashboard for every other dish in every other city.

Worked solution and explanation

The trap

This is per-key anomaly detection dressed up as a leaderboard. Trending is not 'top-N orders right now'; it is 'a dish whose last 30 minutes run well above its own normal for this hour and weekday'. Anyone can draw a queue into a stream processor. What separates candidates is the baseline lookup and the partition key: sort by absolute volume and the same popular dishes trend every night, key the stream badly and one viral dish stalls trending for every city.

The default reach is a streaming job that counts orders per dish over 30 minutes and sorts by volume. Pad thai always 'trends' because it always sells; the small dish that just tripled never shows. When a social post drives one dish to many times its normal rate, a poorly keyed job lets that hot key drag every worker, and the view for every other city goes stale.

Walk the requirements

Step 1: Get order events to the trending view within minutes

Live orders land on city_dish_keyed_order_queue, which trending_scorer_30_min_rate_per_city_dish reads continuously to keep the last-30-minute rate for each city and dish pair. The scorer writes into trending_store, and partner_dashboard reads from there within a few minutes. Without a streaming scorer the view runs on batch cadence and cannot be 'right now'; without the queue, a burst that outruns the scorer either drops orders or blocks the ordering app.

Step 2: Score each dish against its own historical baseline

baseline_batch reads two years of order_history once a day and writes each dish's normal rate per hour and weekday into dish_baseline_store; the scorer looks it up on every order. The score is current_rate / baseline_rate with a noise floor so a dish going from 1 to 3 orders does not headline. A daily refresh is enough because normal drifts slowly; only the current window has to be live.

Step 3: Key by dish so one viral spike stays contained

Keying city_dish_keyed_order_queue by city and dish_id puts each dish's orders on one partition, and the 16 parallel scorer tasks each own a slice of those keys. A viral dish backs up its own partition while every other partition keeps moving, so other dishes and cities update on schedule. Key by order_id instead and every task carries a share of the hot dish's state, so the spike slows everyone.

The shape that fits

Common follow-up questions

• A new dish appears with no history. How does this design score it, and what shows on the dashboard during its first hour? (Tests cold start: with no baseline the ratio is undefined, so the scorer either labels the dish new or falls back to a category-level baseline, without failing on a missing lookup in `dish_baseline_store`.)
• Restaurants want trending dishes per neighborhood, not just citywide. What changes, and where? (Tests whether the candidate sees the key as the lever: the key becomes city, neighborhood and dish, per-key state grows, and the baseline batch has to compute per neighborhood as well.)