Five Times the Traffic, Five Times the Bill

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
hard
Seniority
L7

Problem

Our fixed-size Spark cluster sits idle 80% of the time yet still falls behind whenever a product launch pushes event traffic to 5x its quietest hour. Design the pipeline that replaces it for both the operations on-call team and the analytics dashboards.

Scale up when needed. Do not bankrupt the team.

Business requirements

Worked solution and explanation

Why this problem exists in real interviews

The bill swings 5x with traffic. Whatever you build has to absorb spikes, serve two consumers on different clocks, and not pollute reports with a small fraction of malformed events. The CFO is on this; the design has to reason about cost, not just about throughput.

First instinct is to size the existing cluster for peak (so it doesn't drop events) and call it done. The bill stays high and steady, which is what the CFO didn't want. The other version of the same instinct is to size it for the average and 'we'll handle spikes by retrying', which means events sit in the producer's memory until they time out and get dropped, and analytics quietly under-counts during launches.

Walk the requirements

Step 1: Put a buffer in front of compute so spikes queue instead of overwhelm

Without a buffer, the only place a spike's worth of events can wait is in producer memory, and producers drop them when they time out. With a buffer in front, anything queue-shaped will do (Kafka, Kinesis, Pub/Sub, SQS): the spike sits there while consumers catch up. The cost shape is also different: the buffer is cheap to scale; the compute behind it scales on actual processing rate, not on the producer's burst rate.

Step 2: Run two paths so each consumer gets the cadence it needs

Operations alerts within minutes; analytics dashboards tolerate fifteen. If both consumers read from the same processor, you're either over-spending to keep analytics on the streaming tier or under-serving operations on the batch one. A streaming consumer for ops (Flink, Spark Streaming, Kafka Streams are all viable) and a micro-batch consumer for analytics share the same buffer and run independently.

Step 3: Let elastic compute follow load instead of sitting at peak

Spike at peak, idle the rest of the day. The bill stops swinging when the worker pool grows during the spike and shrinks afterward. Combine that with the buffer above and the worst case becomes 'queue gets deep for ten minutes' instead of 'cluster falls over and on-call comes in.' On the canvas, both processors carry an autoscaling parallelism setting instead of a fixed task count.

Step 4: Route invalid events somewhere other than the analytics tables

A small fraction of events are malformed every hour. They can't reach the analytics tables: that's a data-quality bug analysts will chase for a week. They also can't be silently dropped, because at this volume even a small fraction is enough rows for someone downstream to eventually notice the count is off. A separate location with the original payload and the failure reason is the third option, and the only honest one. On the canvas this is a validation stage with errorAction set to 'dlq', wired to a rejected-events store only engineering reads.

The shape that fits

Kafka absorbs the spike; `ops_stream` and the validated analytics path read it independently on autoscaling compute, and failed events land in `rejected_events`.
  • event_buffer (queue, Kafka)
  • ops_stream (transform, Flink); freshness: < 1min; parallelism: autoscale 2-12 tasks
  • operations_alerts (consumer, alerting)
  • event_validation (quality gate, Great Expectations); on error: dlq
  • rejected_events (storage, object storage)
  • analytics_micro_batch (transform, Spark); freshness: < 15min; parallelism: autoscale 2-12 tasks
  • analytics_store (storage, warehouse)
  • analytics_dashboards (consumer, Metabase)
  • event_buffer -> ops_stream
  • ops_stream -> operations_alerts
  • event_buffer -> event_validation
  • event_validation -> rejected_events
  • event_validation -> analytics_micro_batch
  • analytics_micro_batch -> analytics_store
  • analytics_store -> analytics_dashboards

Common follow-up questions