The Queue That Wouldn't Stop Growing

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Asked in

Dailymotion

Problem

Your streaming video event pipeline shows consumer lag spiking from near-zero to over 500,000 messages within two hours. You need to diagnose whether the cause is a producer burst or a consumer slowdown, then design a monitoring and auto-remediation system that can detect, alert on, and automatically recover from future lag events.

500,000 messages behind and the number keeps climbing.

Business requirements

• Lag climbed past 500,000 messages before anyone noticed; future lag events have to be detected and raise an alert.
• When lag builds up again, the pipeline has to recover on its own instead of waiting for an engineer to intervene.

Worked solution and explanation

What the lag is really telling you

Consumer lag is one number with two opposite causes. This problem is really about building a monitor that can tell those causes apart before anything acts on it. Lag is the gap between what producers have written and what a consumer group has committed, so a producer burst and a consumer slowdown both push it up. The trap is wiring alerting and scaling straight to that single number. A scale-out triggers a rebalance, the rebalance spikes lag by about 80,000, the alert fires, and the scaler adds another pod. On-call learns to ignore the page that finally matters.

Build it

Step 1: Read lag off the bus, not off CPU

lag_monitor takes its signal from event_bus, where each partition's end offset and each group's committed offset both live. That is the only place the diagnosis is possible. In this incident, producers tripled to 240,000 events a second while commit rates stayed flat: a burst. The existing autoscaler watches CPU, which barely moves when a group is short of replicas, so it never reacted.

Step 2: Alert per group, after the rebalance pause

Recommendations pages above 5,000, ad targeting above 50,000, and the lake writer only above one million. One shared threshold either floods the lake channel or misses recommendations. Every new pod pauses its group for 10 to 30 seconds and spikes lag by about 80,000, which is already 16 times the recommendations limit. So the alert rule on lag_monitor fires only when a group stays over its own threshold past that pause.

Step 3: Let an orchestrator drive recovery, with a ceiling

lag_monitor hands the sustained-lag signal to remediation_orchestrator, which scales stream_consumers without waiting for a human. At 240,000 events a second and 8,000 per recommendation replica, recovery needs about 30 replicas. The topic has 48 partitions, and a replica past the partition count sits idle while still forcing a rebalance. Cap scale-out there, and wait out a cooldown before taking the next step.

Scaling on CPU

Reacts to how busy pods are, not to how far behind they are. A group starved of replicas can sit at moderate CPU while lag climbs past 500,000, which is exactly what happened here.

Scaling on sustained lag

Reacts to the gap the business actually cares about, per group, and only after a rebalance pause has had time to clear. It stops at the partition count, where more pods stop helping.

Common follow-up questions

• A real producer burst hits while a scale-out is still rebalancing. What does this design do? (Tests whether the candidate lets the cooldown settle the in-flight scale-out, then scales again only if lag is still over the threshold past the rebalance window.)
• Lag climbs, but producers are steady and commits have dropped. What should the orchestrator do instead of adding pods? (Tests whether the candidate treats a consumer slowdown as a different failure: page with the diagnosis attached, and do not trigger a scale-out that cannot help.)