The What-If Machine
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L5
- Asked in
- Netflix
Problem
What-if questions about fill rate, reach and frequency keep piling up from our analysts, and a single configuration can mean up to 1,000 simulations. Design the data pipeline behind a system that matches ad inventory (slots) against ad campaigns, so users can submit a configuration and explore the results later.
A million slots. A thousand campaigns. Every combination matters.
Business requirements
- A configuration's variants have to be comparable in the same working session, not the next day.
- Losing a whole configuration's worth of work just because one variant blew up isn't acceptable.
- A submitted configuration runs in the background; the caller shouldn't have to keep a browser tab open while it runs.
Worked solution and explanation
The job handle is the product
This is a fan-out with a walk-away user dressed up as an ads simulator. Anyone can write the matching logic. What separates candidates is the contract: the user submits a configuration, leaves, and comes back to compare up to 1,000 variants. Draw one big batch job that loops over variants and writes at the end, and a single bad variant aborts the run and takes every finished variant with it. The failure radius is wrong, not the algorithm.
Walk the requirements
Step 1: Queue one message per variant
Submission writes the variants to a queue and returns immediately. The queue is what lets the caller close the tab, absorbs bursts, and lets a dead worker's message be picked up again.
Step 2: Fan out onto independent workers
A thousand serial 2-minute runs is over thirty hours. Workers pull variants from the queue concurrently, so total time scales with the capacity you pay for, not with variant count. Each worker reads the inventory and campaign data it matches against.
Step 3: Write each variant's result on its own
Results land in object storage as each variant finishes, one object per variant. A failed variant is retried alone, or recorded as failed, and the other 999 stay valid.
Step 4: Keep job status outside the compute
Intake records a job id, and workers update per-variant status in a small relational store. The user checks progress against that store, so a stuck or dead worker never hides what happened.
- config intake (source, API)
- inventory and campaign data (source, PostgreSQL)
- variant queue (queue, Kafka)
- matching workers (transform, Spark); parallelism: auto; on error: retry
- variant results (storage, object storage)
- job status store (storage, PostgreSQL)
- comparison dashboard (consumer, Metabase)
- config intake -> variant queue
- config intake -> job status store
- variant queue -> matching workers
- inventory and campaign data -> matching workers
- matching workers -> variant results
- matching workers -> job status store
- variant results -> comparison dashboard
- job status store -> comparison dashboard
- One batch job
- Time grows with variant count, one failure aborts the run, and the user must wait for the final write to see anything.
- Variant per message
- Time is bounded by worker capacity, a failure retries only its own variant, and finished results are readable immediately.
Common follow-up questions
- If one variant keeps failing its retries, how does the user see what is blocking their configuration? (Probes the failed state in the status store and a bounded retry budget with a dead letter path.)
- What changes when one variant takes thirty seconds and another takes thirty minutes? (Probes worker scheduling so slow variants do not starve fast ones.)