A vendor drops files into SFTP at irregular times

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A vendor drops files into SFTP at irregular times. The pipeline currently has no boundary check; the orchestrator just starts at a fixed time and hopes the file is there. Apply the sensor-placement framing this section just taught and add a sensor at the boundary between the orchestrator and the external SFTP source. The structural footprint of a boundary sensor is a quality_check node (Monte Carlo, Bigeye, Anomalo, or dbt tests) sitting between the SFTP source and the daily Spark transform; the quality_check verifies the file landed correctly and is non-empty before the downstream pipeline proceeds. Sensors earn their place at the orchestrator/external-world boundary; the quality_check is the canvas's representation of that role.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.