Traders, Risk, and the Regulators

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

Energy price ticks peak at 50,000 a second and twelve desks send about 300 trade executions a second between them, each desk feed replaying its last few minutes whenever it reconnects. Design the pipeline that brings our market prices, executed trades and open positions together for the risk managers who watch exposure through the trading day.

Markets move in milliseconds. The pipeline has to keep up.

Business requirements

• Risk managers need executed trades and current market prices on their view within minutes of execution, so nothing on the path to their view can wait for an hourly or overnight load.
• Every number must be auditable: the trade history behind any past day has to be kept and reproducible.
• The data must be reliable: a trade delivered twice must never be counted twice in an open position.

Worked solution and explanation

Two clocks, one source of truth

This is an exactly-once counting problem dressed up as a latency problem. Everyone can draw Kafka into a stream job into a dashboard. The trap is the retry: a producer resends a trade, the position store appends it, and a risk manager sees exposure inflated by a trade that happened once. The hedge placed against that phantom exposure is real money.

The second half is the word auditable. If the only copy of a trade lives in a mutable position table, yesterday is gone the moment the table updates. Land the same deduplicated events in an immutable history store, and the position table becomes a derivative you can rebuild.

Walk the requirements

Step 1: Keep the risk path streaming end to end

Prices and trades arrive on Kafka, a Flink job processes them, and the result lands in the position store the dashboard reads. Every hop is real-time. One hourly or overnight stage anywhere on that path is the hard reject: a risk view fed by a batch load.

Step 2: Dedup and write idempotently before the fork

The execution system stamps every trade with a stable id. The dedup stage keys on it and upserts, so a retry rewrites the same row instead of adding a second one. Name the stage for what it does so a reviewer finds the guarantee on the canvas.

Step 3: Land every event in a table format on object storage

Deduplicated events append to an Iceberg data lake. Snapshots make any past day a query as of that date, so the figures behind it can be rebuilt without touching the live store.

Two feeds, two dedups

Each consumer cleans the raw stream its own way. The position store and the history drift, and nobody can say which one the regulator should believe.

One dedup, one fork

A single stage dedups on the trade id, then fans out. The live view and the replay are derivatives of the same events, so they cannot disagree.

Common follow-up questions

• Regulators ask for last Tuesday's trade report. What do you add, and what stays the same? (Tests whether the candidate adds a batch consumer of the lake without changing the streaming path or the history itself.)
• The streaming job is down for ten minutes in a volatile hour. What does the position store show when it recovers? (Tests catch-up from Kafka offsets, idempotent upserts making the replay safe, and whether risk is told the view was stale.)