Cascading Failure and Backpressure
A cascade starts with a single slow dependency and ends with failures in systems that were never broken. Underneath it is a queue that fills: whenever records arrive faster than a stage can finish them, the difference piles up somewhere, and every somewhere has a size limit.
Arrival rate minus service rate
Suppose a slow database query drops the tax service's capacity from 3,000 requests per second to 1,200. Orders still arrive at 2,000 per second. The order pipeline's in-memory buffer has no size limit, and its process has memory for around 500,000 records, so the buffer grows by 800 records every second until that memory runs out.
| Minutes since slowdown | Buffered records | Process memory |
|---|---|---|
| 0 | 0 | Empty |
| 2 | 96,000 | 19% full |
| 5 | 240,000 | 48% full |
| 10 | 480,000 | 96% full |
| 10.4 | 500,000 | Full: the process runs out of memory |
The crash makes the next minute worse. The restarted process re-reads everything since its last committed offset, so it starts with a backlog and calls the slow service harder than before it died.
Latency drains pools and retries multiply load
Little's law says the number of requests in flight equals arrival rate times latency. At 2,000 requests per second and 50 ms each, the pipeline holds 100 calls open. When latency climbs to 2 seconds, it needs 4,000. A connection pool of 200 runs dry, and records time out while waiting for a connection they never got.
Retries stack on top. If the HTTP client makes 4 attempts and the task wrapping it also makes 4, a single order can produce 16 calls. At 2,000 orders per second, a struggling tax service can face 32,000 requests per second, more than 10 times its healthy capacity. That load keeps it down even after its own fix ships.
3 ways to bound the damage
| Tactic | What it does | What it costs | Use when |
|---|---|---|---|
| Backpressure | A full buffer stops the consumer pulling; records wait in the durable log | Lag grows; nothing is lost while lag stays within retention | The source is durable and the slowdown ends within retention |
| Load shedding | Low-priority records are deferred or dropped so critical ones keep flowing | Deferred work runs later; dropped work is gone | Some consumers can wait and others cannot |
| Degraded fallback | A cached or estimated answer replaces the slow call, flagged as estimated | Results stay approximate until reprocessed | An approximate answer now beats an exact one later |
Backpressure holds only if every stage between the slow service and the durable source honors it. A stage with an unbounded buffer absorbs the pressure, hides it from the stages before it, and becomes the stage that runs out of memory. The order log keeps 72 hours of events, so through a full tax outage, backpressure loses nothing for the first 72 hours. From hour 72 onward, each further hour of outage expires the oldest hour of unread orders.
Shedding needs a priority on every record, set at design time. Payment events feed the ledger and cannot wait; cart-edit events feed only a marketing attribution table and can wait a day. Under slowdown, the pipeline writes cart edits to a deferred table and spends the tax service's 1,200 requests per second on payment events first.
Classification picks the tactic. Timeouts on every call are transient systemic, so the breaker opens and backpressure keeps records in the log. Routing those timeouts to the DLQ would turn a slowdown into millions of good orders awaiting replay.
- Do
- Cap every in-memory buffer and stop pulling when it is full
- Retry at a single layer, inside a retry budget
- Tag each record with a priority before the pipeline needs to shed
- Alert when source lag passes half of the log's retention
- Avoid
- Let any stage buffer without a size limit
- Stack client, task and orchestrator retries on the same call
- Send records to the DLQ because a dependency was slow
- Shed records without logging what was dropped
The tax service goes fully down. The breaker opens, every buffer is capped, and backpressure holds orders in the log, which keeps 72 hours of events. The outage lasts 80 hours. What happens to the orders?
- None are lost, because backpressure keeps every unread order in the durable log
- All 80 hours of orders are lost, because the open breaker discarded each one
- The capped buffers overflow and the pipeline crashes, losing what was in memory
- The first 8 hours of orders expire from the log before the pipeline can read them
About this lesson
This section is part of the Failure Handling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.