File Drop Ingestion

A file drop is a push with the slowest rhythm. A partner, a bank or another team writes files to a bucket or shared folder you can read, on a schedule they control. Your job notices the files, checks them, loads them, and remembers that it did. Each step can go wrong, and the fixes are conventions you agree with the partner before the first file lands.

The half-written file

Uploading a 2GB file takes minutes. A job that lists the folder at 02:03 and finds settlements_2026-10-22.csv cannot tell whether the upload finished or whether the file holds the first 1.3GB of 2. Loading it produces a row count that passes any loose check and a daily total short by a third.

Two conventions solve this; ask the partner for at least 1. With write-then-rename, the partner uploads to settlements_2026-10-22.csv.tmp and renames it when the upload ends, and your job ignores anything ending in .tmp. With a sentinel, an empty file named _SUCCESS by convention is written only after every data file in the batch has landed; if it is missing, your job exits and tries again on its next schedule.

The manifest

A sentinel says the batch is complete. A manifest says what complete means: a small file listing every data file in the batch with its row count and a checksum. With a manifest your job can prove it received the batch the partner sent, rather than whatever happened to be in the folder when it looked.

{
  "batch_date": "2026-10-22",
  "generated_at": "2026-10-23T01:47:12Z",
  "files": [
    {"name": "settlements_2026-10-22_part1.csv", "rows": 184220, "sha256": "9f2c7a..."},
    {"name": "settlements_2026-10-22_part2.csv", "rows": 184301, "sha256": "41be03..."}
  ],
  "total_rows": 368521
}

def load_batch(folder, store):
    if not folder.exists("_SUCCESS"):
        return "waiting"
    manifest = folder.read_json("manifest.json")
    checksums = [f["sha256"] for f in manifest["files"]]
    if store.already_loaded(manifest["batch_date"], checksums):
        return "already loaded"
    all_rows = []
    for entry in manifest["files"]:
        name = entry["name"]
        if not folder.exists(name):
            raise ValueError(f"missing file: {name}")
        if folder.sha256(name) != entry["sha256"]:
            raise ValueError(f"{name}: checksum does not match manifest")
        rows = folder.read_csv(name)
        if len(rows) != entry["rows"]:
            raise ValueError(f"{name}: expected {entry['rows']} rows, got {len(rows)}")
        all_rows.extend(rows)
    store.replace_partition("raw.settlements", manifest["batch_date"], all_rows)
    store.record_loaded(manifest["batch_date"], checksums)
    return "loaded"

A missing file, a checksum that differs from the manifest, or a row count that differs from it raises and blocks the load: the manifest and the data disagree, and loading either version in silence would be wrong. Replacing the whole batch_date partition makes a rerun safe. Recording the checksums means a partner who re-uploads yesterday's batch, sentinel and all, triggers a run that reads the manifest and stops.

Defensive validation

Check	What it catches	Action
Sentinel present	A batch still uploading	Wait, retry next schedule
Every manifest file exists in the folder	A file lost in transfer	Block, alert the partner
Row count equals the manifest	A truncated or half-written file	Block
Header matches expected columns	A renamed or reordered column	Block on rename, warn on an added column
File is not empty	The partner's export produced nothing	Block
Checksums not seen before	A re-upload of an old batch	Skip and log

The header check is the schema drift lesson applied to files. A CSV carries its column names in line 1. A partner who adds currency at the end has made a backward-compatible change you accept with a warning; one who renames amount to amount_cents has broken the contract, and you stop and talk before loading a row of the new shape.

A bank drops 3 settlement files nightly plus _SUCCESS. On Thursday the sentinel appears and the manifest lists 3 files, but only 2 are in the folder, and the row counts of those 2 match the manifest. What should the loader do?

Load the 2 files that match and leave the partition short, since the sentinel proves the bank considers the batch finished.
Wait for the third file, because the bank's transfer is still in progress and the file will appear within the next schedule.
Load the 2 files and record the batch as loaded, so Friday's run picks up the third file as part of Friday's batch.
Block the load and alert: the sentinel says the batch is complete, but the manifest and the folder disagree, so a file was lost after the bank's export finished.

About this lesson

This section is part of the Ingestion Patterns: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.