Backpressure at the Ingestion Layer

Backpressure travels upstream from a slow stage to a faster one until it reaches the pipeline's edge. Ingestion is that edge, and the next hop upstream is a system you do not run. When arrivals outpace what ingestion can drain, you have 3 possible responses. Which of them are open to you depends on the source.

Classify the source before choosing

SourceCan it wait?Can it replay?Response
Partner order webhooksYes, retries for a limited windowWithin that windowBuffer; return 429 near full
Mobile clickstreamNo, the app discards what it can't sendNoBuffer, then sample under pressure
Internal payment eventsYes, rows wait in its outboxYes, from the outboxSignal; the outbox is the buffer
Vehicle location pingsNoNoKeep the latest per vehicle, drop the rest
Log-based CDCYes, but the database holds its log meanwhileWithin log retentionDrain fast into your own log; slow down after it

Sizing the buffer you depend on

A worked surge: traffic normally runs at 2,000 events per second and workers drain 3,000. A promotion pushes arrivals to 12,000 per second for 15 minutes, with events averaging 1 KB. During the surge the backlog grows by the gap between arrivals and drain; afterwards it shrinks only by the spare drain capacity.

Buffer capacity must cover the longest planned surge, with margin. The peak lag must fit the freshness SLA, and if it does not, a bigger buffer cannot help: the fix is more drain capacity or a drop rule for part of the stream. Recovery time depends on spare capacity, so doubling the workers' headroom halves it.

When backpressure reaches the database

Slowing a CDC reader looks harmless because the source can wait, but the database waits by keeping every log segment the reader hasn't confirmed. A reader stalled for 6 hours on a primary writing 20 GB of log an hour pins 120 GB of disk, and a full disk stops the primary taking writes: ingestion has taken down checkout. So the CDC reader never absorbs backpressure: it copies changes into a durable log you own as fast as the database produces them, and every slowdown happens after that log. Track the log bytes retained for the reader, with a ceiling past which you drop it and re-snapshot in chunks.

Webhooks and clickstream pass the admission rule. The CDC reader writes straight to the durable log, so the database never waits on a slow drain.
  • partner webhooks (source, HTTPS)
  • mobile clickstream (source, app SDK)
  • orders CDC (source, log reader)
  • admission (quality gate, ingest API); on error: 429 above 95% fill
  • durable log (queue, broker); parallelism: 48 partitions
  • ingest workers (transform, workers); parallelism: 3,000 events/s; backoff: exponential with jitter
  • raw tables (bronze, lakehouse); freshness: 1 hour
  • partner webhooks -> admission
  • mobile clickstream -> admission
  • admission -> durable log
  • orders CDC -> durable log
  • durable log -> ingest workers
  • ingest workers -> raw tables
  • Sources: partner webhooks, mobile clickstream, orders CDC
  • Edge: admission
  • Buffer: durable log
  • Drain: ingest workers, raw tables

An admission rule that uses all 3

This handler fronts a shared ingestion endpoint. Below 70% fill it accepts everything. Between 70% and 90%, loss-tolerant events are sampled down to 1 in 10, and above 90% they are all dropped. Must-keep events are accepted until 95%, after which producers get 429 with a Retry-After header.

def admit(event, buffer, dropped):
    fill = buffer.size() / buffer.capacity
    if event.loss_tolerant:
        sampled = event.sample_key % 10 == 0
        if fill < 0.70 or (fill < 0.90 and sampled):
            buffer.put(event)
        else:
            dropped[event.event_type] += 1
        return 202, {}
    if fill < 0.95:
        buffer.put(event)
        return 202, {}
    return 429, {"Retry-After": "30"}

Dropped events still get 202, because a producer told to retry something you chose to discard only adds load. Every drop is counted by event type, since an uncounted drop shows up as a fake traffic decline.

A fleet of 50,000 scooters sends GPS positions every 2 seconds with no retries. The only consumer is a live map of each scooter's current position. The shared buffer nears full during a surge. Which response fits?

About this lesson

This section is part of the Ingestion Patterns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.