The Decision Before the Door Closes

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L5

Asked in

RBC

Problem

What pipeline scores millions of card transactions a day for fraud before each authorization completes, well under a second after it starts? When scoring is unavailable, a fallback still decides every transaction: small ones go through and large ones are blocked. Chargebacks confirm fraud weeks later and have to reach the weekly model retrain.

The window to stop it is smaller than you think.

Business requirements

• Authorization completes in well under a second and the fraud score has to fit inside that budget; otherwise we either delay the customer or skip fraud.
• If the scoring service is unavailable we still have to return an approve or block decision by a documented fallback policy; that fallback has to be a real path in the pipeline, not something handled by hand during an incident.
• The model retrains weekly and quality silently degrades without confirmed fraud labels feeding back; chargebacks have to reach training.

Worked solution and explanation

What this really is

This is a race against a deadline dressed up as fraud detection. Anyone can draw a queue feeding a streaming scorer. What separates candidates is where the fallback hangs. If it sits downstream of the scorer, it only sees transactions the scorer already handled, so the moment the scorer is down or slow, nothing reaches the fallback and no transaction gets decided. The second trap is quieter: chargebacks land weeks later in a place the model never reads, and the weekly retrain keeps learning from its own guesses.

Building it

Step 1: Put a queue between the gateway and scoring

Card transactions publish to a Kafka topic keyed by card. The gateway never calls the scorer directly, so a slow model cannot hold an authorization thread open. The scorer and the fallback branch both hang off this one queue.

Step 2: Score on a streaming path with a hard deadline

A Flink fraud scorer set to real-time reads the queue and hands its decision to auth decision inside its slice of the budget. A score that misses the deadline counts as unavailable. A late answer is treated like no answer, never as a reason to keep the customer waiting.

Step 3: Route outages and timeouts to the fallback

Two error-path edges feed fallback policy: one from the queue when the deadline expires, one from the scorer when it fails. The policy approves small amounts and blocks large ones, then hands its decision to the same auth decision consumer. Keep the threshold in versioned config, not in code someone edits at 3am.

Step 4: Close the label loop

Both the scorer and the fallback write to a decision log, so every authorization is on record. Chargebacks arrive weeks later through chargeback feed and land in a warehouse-backed labels store next to those decisions. The weekly retrain reads that join, so it learns from confirmed outcomes and not from its own predictions.

Fallback in series

Scorer, then fallback, then auth. Every decision passes through the fallback, but its only input is scorer output. If the scorer dies, the fallback idles and authorizations hang or drop.

Fallback on its own branch

The fallback reads the raw transaction from the queue on timeout and catches scorer errors. It needs only the amount, so an outage lowers decision quality but never stops decisions.

Common follow-up questions

• During a scoring outage, the fallback approves a just-under-threshold charge that turns out to be fraud. Where does the post-mortem look? (Tests whether the `decision log` records the path and policy version, so the chargeback is attributed to the threshold rather than the model.)
• Should fallback decisions be used as training rows in the weekly retrain? (Tests label hygiene: fallback outcomes reflect a threshold rule, not the model, so they need a flag or should be excluded.)
• The same chargeback arrives twice from the card network. What stops it from counting as two fraud labels? (Tests deduplication on dispute id before the `labels store`.)