A partner drops daily CSVs into an SFTP inbox
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
Interview Prompt
A partner drops daily CSVs into an SFTP inbox. The current pipeline lists the directory and processes every file every run, which means yesterday's files re-process every day. Worse, the partner sometimes re-uploads yesterday's file with the same name and different contents, occasionally writes half-files (sentinels) the pipeline starts reading, and switches encoding mid-quarter without notice. The section's pattern is the manifest: track which paths have been processed, validate filename pattern and encoding before reading, skip sentinels. Process the new file by adding a manifest store, a validation transform whose name states the checks (filename pattern, size, encoding, header row, sentinel skip), and a processed-prefix sink so the inbox only contains unprocessed files.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision