The Signals That Power Recommendations

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Asked in

Stitch Fix

Problem

A recommendation platform serving millions of users learns from clicks, purchases, saves, and ratings emitted by several product surfaces, and different teams consume those signals with different latency needs. Design the ingestion and feature pipeline that keeps user profiles current.

Fresh signals, many teams, one pipeline.

Business requirements

• The recommendation API uses 'recently viewed' features; if those are stale, recommendations get noticeably worse.
• Six product teams emit signals in their own formats with no shared schema today; consumers can't keep adding format-specific code each time a producer ships a change.
• Under GDPR, a deletion request has to remove the user from the feature store, the historical archive, and any derived features.

Worked solution and explanation

What this really is

This is a contract problem dressed up as an ingestion problem. Six teams, six formats, a recommendation API that goes stale in minutes, and a GDPR clock. Anyone can draw Kafka feeding a feature store. What separates candidates is whether the shape is fixed at publish or repaired downstream, and whether deletion is part of the design or a later project. Harmonize downstream and every consumer carries six parsers, so one producer change breaks all of them. Bolt deletion on later and the user survives in derived stores nobody listed.

Walk the requirements

Step 1: Gate every publish on the canonical shape

The schema contract sits in front of the event bus and rejects nonconforming events. The team that broke the shape learns at publish time. Without the gate, consumers find out in production.

Step 2: Split fresh features from heavy ones

'Recently viewed' goes through a Flink job into the online feature store within a minute. Long-window aggregates run in Spark over the archive. Streaming every feature costs a lot and buys nothing. Batching every feature means stale recommendations.

Step 3: Make deletion an event with receipts

A deletion request rides the same bus. The deletion orchestrator removes the online row, deletes the user's events from the archive, and triggers a recompute of the affected batch features. It collects a confirmation from each store and alerts when one is missing inside the window. You answer the regulator with those receipts, not a promise.

The reference design

Common follow-up questions

• A producer adds an `event_type` value that no consumer reads yet. What does the schema contract enforce, and what does it let through? (Checks that you know the difference between a new value inside the shape and an additive, optional change to the shape itself.)
• One store's deletion confirmation is overdue. What does the deletion orchestrator do, and what goes in the audit response? (Checks for retries and an alert, and a per-store status that shows confirmed and outstanding stores rather than a single pass or fail.)