# A partner drops daily CSVs into an SFTP inbox

Canonical URL: <https://datadriven.io/problems/a-partner-drops-daily-csvs-into-an-sftp-inbox-the-current-p-876ae632>

Domain: Pipeline Design · Difficulty: medium

## Problem

A partner drops daily CSVs into an SFTP inbox. The current pipeline lists the directory and processes every file every run, which means yesterday's files re-process every day. Worse, the partner sometimes re-uploads yesterday's file with the same name and different contents, occasionally writes half-files (sentinels) the pipeline starts reading, and switches encoding mid-quarter without notice. The section's pattern is the manifest: track which paths have been processed, validate filename pattern and encoding before reading, skip sentinels. Process the new file by adding a manifest store, a validation transform whose name states the checks (filename pattern, size, encoding, header row, sentinel skip), and a processed-prefix sink so the inbox only contains unprocessed files.

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/a-partner-drops-daily-csvs-into-an-sftp-inbox-the-current-p-876ae632)
- [System Design Interview Questions](https://datadriven.io/data-engineering-system-design)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). 100% free data engineering interview prep. Live code execution against Postgres 16, Python 3.11, and Spark sandboxes. No paywall, no premium tier, no signup gate.