What Should We Recommend Tonight

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L6

Problem

Orders and menu updates live in separate operational systems that are never joined, yet the recommendations team needs a feature store it can query in real time. At our meal kit service, those systems hold customer order history and the full menu catalog with nutritional and ingredient data. Design the ingestion and feature pipeline.

They ordered pad thai twice. That means something.

Business requirements

• The recommendation model runs at checkout and customers are waiting; the result has to feel instant.
• When a menu item gets a new allergen, recommendations to allergic customers have to reflect that change quickly; an unsafe recommendation is a real problem.
• When the model retrains, training rows for an old order have to use the menu and ingredients as they were on that order's date, not as they are now.
• When 10% of users are on a new model variant, both variants have to read the same feature values for the same user; otherwise the experiment is biased.

Worked solution and explanation

What this really is

This is a dual-store feature platform with a food-safety clock bolted on. Anyone can draw orders and menu flowing into one table the model reads. The real test is seeing that one store cannot serve two masters: checkout needs today's value in tens of milliseconds, and training needs the value as it was on each order's date. Collapse them into a nightly join and three things break together. An allergen added at noon keeps being recommended to allergic customers until tomorrow. Training joins today's menu onto last year's orders and learns the future. Checkout waits on a warehouse query.

Walk the requirements

Step 1: Precompute for checkout

Customer and item features live in online_feature_store, keyed for point lookups. The service fetches them, the model scores, and the page renders inside its budget. Computing features at request time is the design where checkout hangs at dinner rush.

Step 2: Stream the menu, because allergens are a safety clock

menu_stream tails menu CDC and upserts the item row in the online store within a minute. The same event lands in offline_feature_store with its effective date. A nightly refresh here is not a freshness tradeoff; it is an unsafe-recommendation window.

Step 3: Join training rows as of the order date

Training reads each historical order against the menu state on order_date, not the latest row. That only works if the offline store kept every version, which is why the stream appends history there instead of overwriting.

Step 4: Read features once per request

recommendation_service fetches the user's features once and hands the same values to whichever variant scores. Two variants with two reads, or two caches, means the experiment measures cache timing as well as the model.

One nightly table

Checkout queries the warehouse. A noon allergen change waits for tomorrow's run. Training joins the current menu onto old orders, so offline metrics beat production.

Online plus offline store

Checkout hits online_feature_store in milliseconds. menu_stream pushes allergens in under a minute. training_job reads offline_feature_store as of each order_date.

Common follow-up questions

• A new feature needs a heavy nightly computation. How does it still serve at checkout latency? (Freshness is per feature: batch computes it, writes `online_feature_store`, and reads stay milliseconds.)
• `menu_stream` is down for 20 minutes during an allergen update. What do you want to happen? (Tests whether the candidate alerts on stream lag and fails safe by suppressing affected items rather than serving stale rows.)