Risk Models on Week-Old Data

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain: Pipeline Design
Difficulty: medium
Seniority: L5

Problem

Credit models are running on week-old data for the risk team, even though decisions happen in real time and repayments stream in continuously across thousands of applications a day. Design a Kafka-based pipeline that delivers loan application events, credit decisions, and repayment data to the risk analytics layer within minutes of occurrence.

Loan approved. Loan denied. Every decision is an event.

Business requirements

Risk managers can't act on a sharp shift in approval rate when they're working from week-old data.
The daily sum of repayments has to reconcile against the loan management system's ledger; finance can't sign off if the numbers drift.
Regulators may ask for the full lifecycle of any loan within a day of asking; every state change has to be retrievable.

Worked solution and explanation

Why this problem exists in real interviews

Risk wants minutes-fresh data; finance wants the daily total to match the ledger; regulators want any loan's full lifecycle retrievable on a day's notice. Three consumers reading the same loan events from three different angles. The trap is one streaming pipeline that satisfies risk and silently misaligns the other two.

The default shape is Kafka into a stream processor that updates a risk dashboard, with a side write to a database for finance. Risk is happy. Finance asks why the daily repayment total doesn't match the loan ledger and discovers the side write was append-only and missed retries during a connector restart. A regulator asks for the lifecycle of a loan from six months ago and the team can reconstruct most events but not all because nothing was archiving authoritatively.

Walk the requirements

Step 1: Decisions and repayments to risk in minutes

Loan applications, credit decisions and repayments flow from the Kafka topics through a stream processor that updates the risk view, so end-to-end is sub-minute. Risk reads from a stream-fed store, not from a weekly batch. Without a streaming path the risk team stays on week-old data. The stream tech can be Flink or Kafka Streams; what matters is risk seeing decisions within minutes of when they happen.

Step 2: Daily reconciliation against the loan ledger

Finance signs off only if the daily repayment total matches the loan management system's ledger. A reconciliation stage takes two inputs: the archive's repayments and the ledger's own figures. It runs daily, compares the totals and pages with the per-loan diff when they drift. The reconciliation is part of the pipeline, not a manual end-of-day exercise. Without it, drift accumulates undetected and finance loses trust in the numbers.

Step 3: Durable loan archive so regulators can retrieve any loan

Every state change writes to a durable loan-event archive in object storage, partitioned by date and keyed by loan id. The archive is the source of truth: when a regulator asks for a loan's lifecycle, a lookup against it returns every event in order. Without a durable archive the regulator's question becomes a forensic reconstruction. With the archive, the answer is a per-loan query well within the regulator's window.

The shape that fits

Kafka topics feed a Flink job that keeps the risk view fresh within seconds. The same events land in an object-storage archive that regulators read per loan. A daily reconciliation against the ledger pages on drift before finance signs off.

application_events (source, Kafka)
decision_events (source, Kafka)
repayment_events (source, Kafka)
loan_ledger (source, API)
risk_stream (transform, Flink)
risk_view (storage, PostgreSQL)
risk_dashboard (consumer, Grafana)
loan_event_archive (storage, object storage)
daily_repayment_reconciliation (quality gate, dbt tests); alerting: page if daily repayments drift from ledger
finance_signoff (consumer, Metabase)
regulator_loan_lookup (consumer, API)

application_events -> risk_stream
decision_events -> risk_stream
repayment_events -> risk_stream
risk_stream -> risk_view
risk_view -> risk_dashboard
application_events -> loan_event_archive
decision_events -> loan_event_archive
repayment_events -> loan_event_archive
loan_event_archive -> daily_repayment_reconciliation
loan_ledger -> daily_repayment_reconciliation
daily_repayment_reconciliation -> finance_signoff
loan_event_archive -> regulator_loan_lookup

Common follow-up questions

A connector restart causes some repayment events to replay. Where would you make writes idempotent so finance doesn't double-count and risk doesn't drift? (Tests whether the candidate places idempotent writes keyed on event id at both the archive and the risk view. Replays re-emit events, and without idempotency at both sinks finance double-counts and risk drifts.)
The reconciliation fires because the pipeline total is over the ledger by a small amount. How do you tell whether it's a pipeline bug or a ledger bug? (Tests whether the candidate uses the per-loan diff the reconciliation produces. A loan that appears several times in the pipeline points to a duplicate bug in the pipeline. A loan missing from the ledger points to a problem at the source.)