Mark to Market

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

mid

Asked in

Fidelity

Problem

About 2 billion trade execution events a day flow through a brokerage platform, and both the risk desk and regulatory reporting depend on that single feed. Design the pipeline that takes those trades in and serves both teams from it.

Business requirements

• Risk wants position dashboards that stay within a few seconds of the market during trading hours.
• Regulatory reporting needs every trade counted exactly once in the end-of-day books.
• Trades can arrive out of order and some are corrected hours after execution.
• The two teams have different correctness budgets and should not block each other.

Worked solution and explanation

Two correctness budgets, one log

This is a lambda architecture dressed up as a brokerage dashboard. The skill being probed is whether you can fan one replayable log out to two paths with opposite contracts. Risk forgives a position that is off by one fill for a few seconds. Regulators do not forgive a fill counted twice. Anyone can draw a stream processor. The trap is letting one aggregate serve both readers. It is never fast enough to feel live or exact enough to file, and a producer retry or a late amendment lands in the filed books as a reportable error.

The obvious answer is a stream that aggregates fills into a positions table, with the end-of-day report taken as a snapshot at the close. Two facts in the prompt break it: fills arrive out of order, and some are amended or busted hours later. A correction after the close never reaches the snapshot, and a retry double-counts a fill.

Building it

Step 1: Land every fill on a replayable log

Out-of-order arrival and late corrections mean no downstream store can be the source of truth, because it only reflects what arrived before it ran. A Kafka topic keyed by execution id lets the stream and the batch read the same events independently, and lets the batch replay a whole trading day on demand.

Step 2: Stream intraday positions into a serving store

Flink aggregates fills in near real time into Redis, and the risk dashboard reads Redis. A position briefly off by one trade hurts nobody on the desk. Making this path exact is what would make it slow, and it still would not be safe to file.

Step 3: Dedup and reconcile in a daily replay

After the close, a Spark job rereads the full day off the log, anchors aggregation on execution time rather than arrival time, dedups on the stable execution id and applies amendments and busts against the original id. Because it replays, a late fill still lands in the right trading day.

Step 4: Upsert the settled books into the warehouse

An Airflow-scheduled daily task writes the reconciled result into the regulatory warehouse, keyed on execution id. A rerun overwrites rather than appends, so restating a day never double-counts. Regulatory reporting reads only this store, never the Redis positions.

Common follow-up questions

• A correction arrives three hours after the books were submitted. What does the design do? (Whether the books are restateable: rerun the replay for that day and file an amendment with an audit trail, rather than editing in place.)
• Volume doubles and the intraday path lags at the open. What changes, and does it touch the books? (Scaling the stream with partitions and parallelism while seeing that the batch path is decoupled and unaffected.)