Push from Queues and Webhooks

In push ingestion the source does the moving. An event happens, the source hands it to a queue or sends it to an endpoint you run, and your consumer sees it seconds later. You gain freshness and give up control: events arrive whether or not your consumer is healthy, in bursts the source decides, and sometimes more than once.

Reading from a queue: the offset

A queue keeps events in order and numbers them. Your consumer reads from a position called the offset, processes what it read, and commits a new offset so a restart resumes from the right place. Where that commit sits relative to the storage write is the question the pull section asked about its mark, and the answer is the same.

Order of operations	Crash between the 2 steps	After restart
Commit offset, then write to storage	Offset moved, write never happened	Events lost, and nothing records that they existed
Write to storage, then commit offset	Write landed, offset did not move	Events replayed and land a second time

Replay is the one you can live with. The consumer writes each event under its event_id with a replace, so a replayed event overwrites itself and the table ends up identical. Loss leaves a hole only the source can fill, and a source that pushed an event once may have no way to send it again.

def consume(queue, sink, batch_size=500):
    while True:
        batch = queue.poll(max_records=batch_size)
        if not batch:
            continue
        rows = [to_row(event) for event in batch]
        sink.replace_by_key(rows, table="raw.charge_events", key="event_id")
        queue.commit(batch[-1].offset + 1)

Webhooks: a push with no queue in it

A webhook is a push over plain HTTP. The vendor sends a POST to a URL you own each time something happens, waits a few seconds for a 200, and retries if it does not get one. The retry schedule is the vendor's: some give up after a few attempts over minutes, others keep trying for days. Either way the vendor keeps no offset for you, so you cannot ask it to resend from a position. Design as if events can be lost, and keep a backfill path such as the vendor's API.

The trap is doing real work inside the endpoint. An endpoint that parses the payload, looks up the customer and writes to the warehouse before replying takes 4 seconds on a normal day and 30 seconds when the warehouse is busy. Against a 10-second timeout, the vendor declares failure on the busy day and retries, and 2 copies of every event pile onto the same slow warehouse. The way out is a queue between the endpoint and the work.

The endpoint verifies the request, enqueues the raw body, and replies in milliseconds. Bursts and duplicates become the consumer's problem, which it already knows how to handle.

Payment vendor (source, HTTPS POST, 10s timeout)
Webhook endpoint (transform, stateless web service); on error: reply 200 only after enqueue
Event queue (queue, durable, ordered)
Consumer (transform, always on); backoff: exponential on sink errors; idempotency: replace by event_id
Raw events (storage, raw.charge_events)

Payment vendor -> Webhook endpoint
Webhook endpoint -> Event queue
Event queue -> Consumer
Consumer -> Raw events

What the endpoint does, and nothing more

Check the vendor's signature header, so a forged request cannot reach the queue.
Write the raw body and a received_at timestamp, which shows how late each event arrived, to the queue unparsed.
Return 200. Parsing, validation and warehouse writes happen in the consumer, where a slow night costs lag rather than duplicates.

Your webhook endpoint for a shipping vendor parses each payload and appends it straight into raw.shipments. Monitoring shows 3% of shipment events appear twice, and the share rises on nights when the warehouse is busy. What is the most likely cause, and the fix?

The warehouse insert sometimes exceeds the vendor's timeout, so the vendor retries. Enqueue the raw payload, return 200 at once, and have a consumer write with replace by the event id.
The vendor sends each event twice by design, so filter the second copy at the endpoint by remembering the last 1,000 event ids in memory.
The queue commits offsets before the write, so events replay on restart. Move the commit after the write and the duplicates stop.
The orchestrator retried the ingestion DAG after a transient failure. Set its retry count to 0 so each payload is processed once.

About this lesson

This section is part of the Ingestion Patterns: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.