The Models Going Stale

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L6

Problem

Account and transaction activity arrives at about 50,000 events per second and feeds roughly 800 features behind 12 loan risk models. Design the pipeline and stores that serve those features to live scoring of 200 applications per second, keep their history for retraining, and record what each score used.

The model is only as good as what you feed it.

Business requirements

• Credit risk scores at loan application and needs activity from minutes ago, while account closure runs nightly and daily is fine, so the same schedule can't serve both.
• Live scoring at application time needs feature lookups fast enough to sit inside the request, so there has to be a store built for low-latency reads.
• The team retrains on what was true at each past decision, so years of historical feature values have to be kept for training, separate from the online serving store.
• Compliance needs to see exactly which feature values produced each decision as a stored record, not a reconstruction after the fact.

Worked solution and explanation

The trap

This looks like a freshness complaint. Underneath, it is a training-serving consistency problem. Everyone draws a fast store for live scoring. What separates candidates is seeing that one store on one schedule cannot carry four jobs: features from minutes ago, nightly features, years of point-in-time history, and a record of every decision. Collapse them and the model trains on today's values for old loans. It looks brilliant offline, it underperforms in production, and compliance gets a reconstruction instead of a record.

Walk the requirements

Step 1: Split the feature computation by cadence

Recent account activity goes through a streaming path (streaming_features on Flink). The slow aggregates go through a batch path (batch_features on Spark). If everything is streamed you are over-engineering. If everything runs nightly you have the exact staleness the prompt complains about.

Step 2: Give each consumer a store shaped for it

online_store holds only the latest value per entity and serves key lookups inside the request. offline_store keeps every historical value with its effective timestamp. Both are written by both paths, so neither tier has its own private definition that can drift.

Step 3: Train with as-of joins

Retraining joins on (entity_id, event_time) and takes the value that was valid at the decision, never the current one. That is the only way the offline metric predicts the production metric.

Step 4: Log the decision at scoring time

scoring_service writes the feature values, the feature version and the model version to decision_log in the same moment it returns the score. An audit is then a query against that log, with no replay needed.

The reference design

One store, nightly

A nightly batch job overwrites a key-value store. Credit risk misses the last day of activity. Training joins current values onto old events, and the audit trail is rebuilt from application logs.

Two stores, two cadences

Streaming and batch paths share one definition. online_store serves the latest value, offline_store keeps the history for as-of joins, and decision_log stores what was actually scored.

Common follow-up questions

• A new feature needs an aggregation too slow to stream. How does credit risk still use it? (Tests per-feature cadence: it lands through the batch path with a known staleness budget, while the model keeps its real-time features.)
• A bug corrupts values in `online_store`. What do you recover from? (Tests seeing `offline_store` as the source of truth: replay it into the online tier, and use `decision_log` to scope which decisions were affected.)