Six Million Rows Before the Market Opens
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- L5
Interview Prompt
A third-party vendor drops a CSV of 6 million time series price rows over SFTP at 6am each morning, and the quant team's pre-market models run at 8am, so the load has to be queryable in the warehouse by 7:30 with an alert raised if the run looks like it will miss that window. The vendor's file is dirty, carrying duplicate rows, null prices, and malformed timestamps that can't be allowed to reach the models, and the morning load is occasionally rerun on the same file, which must never leave duplicate rows behind. Design the ingestion architecture from the vendor drop through to the quant query surface.
Summary
One massive CSV. Millions of timestamps.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision