Backpressure
Recognizing the question
It arrives as a graph described in words. 'Consumer lag on the orders topic has been climbing since 09:00 and is now at 40 million events.' Or as a design probe: 'the API ingests 10,000 events per second but the enrichment service handles 6,000, what do you do?' The interviewer is testing whether you treat the gap as arithmetic before treating it as an incident. A shortfall of 4,000 per second is 14.4 million events of lag per hour, and the first sentence of a strong answer says so, because it tells the room how long you have before the buffers fill.
- 4.4 hours
- 1.67x
The 4 places the pressure can go
Backpressure is the question of who absorbs the gap, and there are only 4 answers: the buffer absorbs it, the producer slows down, the consumer speeds up, or some data is dropped. Every strategy you have heard of is one of these 4 with a specific mechanism attached. The judgement the interviewer scores is matching the answer to what the data is worth and how long the burst lasts.
| Where the gap goes | Mechanism | What it costs | Fits when |
|---|---|---|---|
| Buffer | log retention on disk, bounded in-memory ring | latency grows with lag, storage | the burst is short and every event must survive |
| Producer slows | blocking send, credit-based flow control, HTTP 429 with Retry-After | upstream latency, upstream memory risk | the producer can hold state, such as a batch extractor |
| Consumer speeds up | more partitions and instances, larger fetch batches, async I/O | money, rebalance pauses, capped by partition count | the rate increase is sustained |
| Drop or sample | shed oldest, keep 1 in N, priority lanes | data loss, biased metrics | telemetry where the trend matters more than each point |
The number that decides it
Parallelism in a partitioned log is bounded by partitions. A topic with 12 partitions can feed 12 consumer instances and no more, so a 13th instance sits idle. If the enrichment service needs 10 instances at 1,000 events per second each to keep pace, and the topic has 8 partitions, scaling consumers does nothing until the topic is repartitioned, which is a producer-side change with its own key-ordering consequences. State the partition count before proposing more consumers, because the interviewer has often chosen a scenario where that count is the trap.
Bounded versus unbounded
Unbounded buffers convert a throughput problem into a memory problem that surfaces later and worse. The consumer that reads as fast as it can into an in-process queue runs for 20 minutes, then dies with an out-of-memory error, loses its uncommitted offsets and replays, which is where the previous section pays for itself. The strong answer names a bound and says what happens there. Blocking the reader and spilling to disk both work, as does rejecting with a status the producer understands. A broker with retention is a bounded buffer whose bound is measured in hours of disk, and you should know that number for any pipeline you describe.
- checkout_api (source, order events)
- orders_topic (queue, partitioned log); freshness: retention 72 h; parallelism: 24 partitions
- enrich (transform, stream job); parallelism: 24 instances; alerting: lag growing for 10 min
- dead_letter (queue, DLQ); on error: route after 3 attempts
- orders_curated (storage, curated table)
- checkout_api -> orders_topic
- orders_topic -> enrich
- enrich -> dead_letter
- enrich -> orders_curated
What a top answer sounds like
Open with the arithmetic, name the deadline from retention, then pick where the gap goes and defend it with the data's value. Then add the operational layer: alert on lag growth rate rather than lag level, because a lag of 2 million that is shrinking is a recovery and the same lag growing is an incident. Mention that slow consumers are usually slow because of a synchronous call per event, and batching those calls often yields a 5x to 10x improvement before any scaling happens. Finish with the failure mode of your own choice: if you add consumers, say what happens to ordering within a key; if you drop, say which dashboard now reads low.
About this lesson
This section is part of the How Data Moves: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.