The Four Shapes of Ingestion
Ingestion is the first role in any pipeline: the step that copies data out of a system you do not control into storage you do. What changes from source to source is how the data arrives, and that decides how the job is written, how it fails, and how you rerun it.
Who moves first, and what arrives
Two questions sort every source: who initiates the transfer, your job or the source, and what unit shows up. The answers give 4 shapes, and each carries its own way of remembering where it stopped.
| Shape | Who initiates | Unit that arrives | Typical source | How you track progress |
|---|---|---|---|---|
| Pull from a database | Your job | Rows matching a query | Your company's orders database | The largest timestamp you have loaded |
| Push from a queue or webhook | The source | One event at a time | A payment provider sending charge events | The offset you last committed |
| File drop | The source, on its schedule | A whole file or a batch of files | A partner uploading a nightly CSV | The list of files already processed |
| API pull | Your job | A page of records per request | A vendor's web endpoint | The cursor of the last page fetched |
- Orders DB (source, read replica)
- Charge events (queue, durable queue)
- Partner files (source, object storage bucket)
- Ads vendor API (source, HTTPS endpoint)
- Pull job (transform, nightly batch); freshness: by 04:00; idempotency: replace by order_id
- Queue consumer (transform, always on); idempotency: replace by event_id
- File loader (transform, runs when sentinel appears); on error: block on count mismatch
- API crawler (transform, hourly); retries: 5; backoff: exponential, honors Retry-After
- Raw tables (storage, data lake)
- Orders DB -> Pull job
- Charge events -> Queue consumer
- Partner files -> File loader
- Ads vendor API -> API crawler
- Pull job -> Raw tables
- Queue consumer -> Raw tables
- File loader -> Raw tables
- API crawler -> Raw tables
- Sources: Orders DB, Charge events, Partner files, Ads vendor API
- Ingestion: Pull job, Queue consumer, File loader, API crawler
- Raw: Raw tables
Picking a shape for a new source
The source makes most of the choice. If you can run SQL against the system, pull from it. If it emits events or offers webhooks, let it push, and each event arrives seconds after it happened. If a partner can only hand you files, you have a file drop. If a vendor exposes a web endpoint and nothing else, you are writing an API job. The real decision comes when a source offers 2 shapes, such as webhooks plus an API: webhooks give freshness, the API fetches what the webhooks missed, and mature setups run both.
A marketing vendor holds your campaign spend. It offers a web endpoint returning 100 campaigns per request and, on your enterprise plan, a nightly CSV in a bucket you own. The spend feeds a daily report. Which shape do you start with?
- API pull. A web endpoint is always fresher than a nightly file, and 100 campaigns per request means the whole dataset arrives in 1 request.
- Push from a webhook. The vendor sends spend changes as they happen, so you never need to store a progress marker or validate a batch.
- File drop. The vendor already produces a complete nightly snapshot in storage you own, which can be validated whole and reloaded in 1 step.
- Pull from the database. Enterprise plans usually include read access to the vendor's database, and a direct query is the most reliable shape.
About this lesson
This section is part of the Ingestion Patterns: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.