A partner drops daily CSVs into an SFTP inbox

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A partner drops daily CSVs into an SFTP inbox. The current pipeline lists the directory and processes every file every run, which means yesterday's files re-process every day. Worse, the partner sometimes re-uploads yesterday's file with the same name and different contents, occasionally writes half-files (sentinels) the pipeline starts reading, and switches encoding mid-quarter without notice. The section's pattern is the manifest: track which paths have been processed, validate filename pattern and encoding before reading, skip sentinels. Process the new file by adding a manifest store, a validation transform whose name states the checks (filename pattern, size, encoding, header row, sentinel skip), and a processed-prefix sink so the inbox only contains unprocessed files.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.