Six Million Rows Before the Market Opens

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

A third-party vendor drops a CSV of 6 million time series price rows over SFTP at 6am each morning, and the quant team's pre-market models run at 8am, so the load has to be queryable in the warehouse by 7:30 with an alert raised if the run looks like it will miss that window. The vendor's file is dirty, carrying duplicate rows, null prices, and malformed timestamps that can't be allowed to reach the models, and the morning load is occasionally rerun on the same file, which must never leave duplicate rows behind. Design the ingestion architecture from the vendor drop through to the quant query surface.

Summary

One massive CSV. Millions of timestamps.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.