Cascading Failure and Backpressure

A cascade starts with a single slow dependency and ends with failures in systems that were never broken. Underneath it is a queue that fills: whenever records arrive faster than a stage can finish them, the difference piles up somewhere, and every somewhere has a size limit.

Arrival rate minus service rate

Suppose a slow database query drops the tax service's capacity from 3,000 requests per second to 1,200. Orders still arrive at 2,000 per second. The order pipeline's in-memory buffer has no size limit, and its process has memory for around 500,000 records, so the buffer grows by 800 records every second until that memory runs out.

Minutes since slowdownBuffered recordsProcess memory
00Empty
296,00019% full
5240,00048% full
10480,00096% full
10.4500,000Full: the process runs out of memory

The crash makes the next minute worse. The restarted process re-reads everything since its last committed offset, so it starts with a backlog and calls the slow service harder than before it died.

Latency drains pools and retries multiply load

Little's law says the number of requests in flight equals arrival rate times latency. At 2,000 requests per second and 50 ms each, the pipeline holds 100 calls open. When latency climbs to 2 seconds, it needs 4,000. A connection pool of 200 runs dry, and records time out while waiting for a connection they never got.

Retries stack on top. If the HTTP client makes 4 attempts and the task wrapping it also makes 4, a single order can produce 16 calls. At 2,000 orders per second, a struggling tax service can face 32,000 requests per second, more than 10 times its healthy capacity. That load keeps it down even after its own fix ships.

3 ways to bound the damage

TacticWhat it doesWhat it costsUse when
BackpressureA full buffer stops the consumer pulling; records wait in the durable logLag grows; nothing is lost while lag stays within retentionThe source is durable and the slowdown ends within retention
Load sheddingLow-priority records are deferred or dropped so critical ones keep flowingDeferred work runs later; dropped work is goneSome consumers can wait and others cannot
Degraded fallbackA cached or estimated answer replaces the slow call, flagged as estimatedResults stay approximate until reprocessedAn approximate answer now beats an exact one later

Backpressure holds only if every stage between the slow service and the durable source honors it. A stage with an unbounded buffer absorbs the pressure, hides it from the stages before it, and becomes the stage that runs out of memory. The order log keeps 72 hours of events, so through a full tax outage, backpressure loses nothing for the first 72 hours. From hour 72 onward, each further hour of outage expires the oldest hour of unread orders.

Shedding needs a priority on every record, set at design time. Payment events feed the ledger and cannot wait; cart-edit events feed only a marketing attribution table and can wait a day. Under slowdown, the pipeline writes cart edits to a deferred table and spends the tax service's 1,200 requests per second on payment events first.

Classification picks the tactic. Timeouts on every call are transient systemic, so the breaker opens and backpressure keeps records in the log. Routing those timeouts to the DLQ would turn a slowdown into millions of good orders awaiting replay.

Do
  • Cap every in-memory buffer and stop pulling when it is full
  • Retry at a single layer, inside a retry budget
  • Tag each record with a priority before the pipeline needs to shed
  • Alert when source lag passes half of the log's retention
Avoid
  • Let any stage buffer without a size limit
  • Stack client, task and orchestrator retries on the same call
  • Send records to the DLQ because a dependency was slow
  • Shed records without logging what was dropped

The tax service goes fully down. The breaker opens, every buffer is capped, and backpressure holds orders in the log, which keeps 72 hours of events. The outage lasts 80 hours. What happens to the orders?

About this lesson

This section is part of the Failure Handling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.