A vendor drops files into SFTP at irregular times
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
Interview Prompt
A vendor drops files into SFTP at irregular times. The pipeline currently has no boundary check; the orchestrator just starts at a fixed time and hopes the file is there. Apply the sensor-placement framing this section just taught and add a sensor at the boundary between the orchestrator and the external SFTP source. The structural footprint of a boundary sensor is a quality_check node (Monte Carlo, Bigeye, Anomalo, or dbt tests) sitting between the SFTP source and the daily Spark transform; the quality_check verifies the file landed correctly and is non-empty before the downstream pipeline proceeds. Sensors earn their place at the orchestrator/external-world boundary; the quality_check is the canvas's representation of that role.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision