Ingestion Patterns

Concepts covered: Change Data Capture, API Ingestion, File-Based Ingestion, Batch vs Streaming, Micro-Batch vs True Streaming

Each of the 4 sources admits 1 ingestion pattern once you ask what it can give. The interviewer is listening for the pattern tied to that answer, because a pattern chosen for the tool you know best reads as a guess, and the follow-ups are built to expose it.

Change data capture from the orders table

Debezium on Kafka Connect reads the Postgres write-ahead log through a logical replication slot and emits 1 event per row change, with the after image and the operation type. The orders table takes 1.2 million inserts and roughly 3 million status updates a day, about 4.2 million change events, each in Kafka under a second after commit. The primary carries 1 slot reading the WAL instead of a polling query scanning updated_at every minute, and deletes arrive as events, which a polling extract never sees.

The cost is operational, and naming it unprompted is what scores. A replication slot pins WAL on the primary's disk until the connector confirms it has read it, so a connector that stops on Friday evening leaves the primary holding every segment written over the weekend, and a full disk takes the production database down with it. Cap the damage with max_slot_wal_keep_size (Postgres 13 and later) and alert on slot lag.

Pulling Stripe through the API

Stripe's REST endpoint pages with a starting_after cursor and returns at most 100 objects per page. Refreshing the full charge history every hour fails within weeks, because the request count grows with the account's age until the job hits the rate limit. Use a high-water mark: store the largest created timestamp from the last successful run, request charges with created[gte] set to it, and page with the cursor so a restart resumes mid-list. That paging only finds new charges, and a charge changes after it is created, when it is captured and sometimes when it is refunded or disputed. Those changes come from /v1/events (charge.captured, charge.refunded, kept 30 days), pulled with the same loop on the event's created, and that feed gives finance its capture dates.

last_hwm = read_watermark("stripe_charges")
cursor = read_cursor("stripe_charges")
max_created = last_hwm

while True:
    page = stripe.Charge.list(
        created={"gte": last_hwm},
        limit=100,
        starting_after=cursor,
    )
    if not page.data:
        break
    write_to_bronze(page.data)
    max_created = max(max_created, *(c.created for c in page.data))
    cursor = page.data[-1].id
    save_cursor("stripe_charges", cursor)
    if not page.has_more:
        break

save_watermark("stripe_charges", max_created)
save_cursor("stripe_charges", None)

Do the division out loud. Stripe's live mode allows 100 read requests a second, so today's 300,000 charges take 3,000 requests, which is around 30 seconds. A 5 million charge backfill is 50,000 requests, roughly 8 minutes if nothing throttles. A 429 gets exponential backoff with jitter, capped near 30 seconds, so 20 workers sharing the limit do not retry together. created[gte] rather than created[gt] re-reads the boundary second on purpose, and the transformation section's merge absorbs the overlap.

Files from the courier partner

The courier partner has no reachable database and no API: they drop around 2 GB of CSV into an S3 prefix at 06:00 UTC, some days as 3 part files. S3 shows each object only once its upload completes, but nothing in a listing says the set for that dt is complete, so an Airflow sensor waits for the partner's _SUCCESS marker: a load that starts after file 1 of 3 parses cleanly and books a third of Tuesday. A day of latency suits a 07:00 report. The failure worth raising first is duplicate delivery: the partner re-uploads Tuesday's file on Wednesday under a new name, and every courier count for Tuesday doubles.

s3://marketplace-lake/bronze/courier_deliveries/
  dt=2026-10-07/
    deliveries_20261007.csv      2.1 GB
    _SUCCESS
  dt=2026-10-08/
    deliveries_20261008.csv      2.0 GB
    deliveries_20261008_v2.csv   2.0 GB   (same bytes, re-uploaded)
    _SUCCESS

A manifest table keyed on each file's SHA-256 checksum skips the duplicate, because the re-upload hashes identically whatever its name. Filename matching misses the rename, and byte size collides whenever 2 exports share a length.

App events, and whether this is a streaming job

The app sends 40 million events a day: 463 a second on average, around 5,000 at the Friday dinner peak. Producers write to a Kafka topic keyed by user_id, so a user's events stay ordered on 1 partition, and 12 partitions at several thousand messages a second each cover the peak and give a consumer group 12 workers. Seven days of retention lets a consumer reset its offset and replay the week, which answers "why Kafka and not a Lambda per event."

Then the question the round is built around: does the operations map need true streaming? It polls every 30 seconds, but operations tolerates data up to 10 minutes stale. A Spark Structured Streaming job committing to bronze every 5 minutes gives 288 commits a day and about 6 minutes of worst-case staleness, inside that bound. Flink with sub-second output would add a second cluster, a second deployment path and a stateful job to operate, for freshness nobody consumes. "Micro-batch at 5 minutes, because operations accepts 10 minutes of staleness" is the batch-versus-streaming answer: the interval comes from the consumer, and the tool follows.

About this lesson

This section is part of the Design a Pipeline: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.