Five Times the Traffic, Five Times the Bill

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
hard
Seniority
L7

Problem

Our platform's data volumes are unpredictable: we see 5x swings between our quietest and busiest hours, with sudden spikes during product launches. We've been running a fixed-size Spark cluster that's over-provisioned 80% of the time and still falls behind during spikes. Operations needs to act on issues within a couple minutes; analytics dashboards tolerate up to 15. A small fraction of incoming events arrive malformed and end up polluting the reports analysts read. The CFO wants the bill to stop swinging with traffic and to come down from where it is today. Design a pipeline that handles variable volume, serves both consumers on the right cadence, keeps bad events out of analytics, and keeps costs predictable.

Scale up when needed. Do not bankrupt the team.

Business requirements

Worked solution and explanation

Why this problem exists in real interviews

The bill swings 5x with traffic. Whatever you build has to absorb spikes, serve two consumers on different clocks, and not pollute reports with a small fraction of malformed events. The CFO is on this; the design has to reason about cost, not just about throughput.

First instinct is to size the existing cluster for peak (so it doesn't drop events) and call it done. The bill stays high and steady, which is what the CFO didn't want. The other version of the same instinct is to size it for the average and 'we'll handle spikes by retrying', which means events sit in the producer's memory until they time out and get dropped, and analytics quietly under-counts during launches.

Walk the requirements

Step 1: Put a buffer in front of compute so spikes queue instead of overwhelm

Without a buffer, the only place a spike's worth of events can wait is in producer memory, and producers drop them when they time out. With a buffer in front, anything queue-shaped will do (Kafka, Kinesis, Pub/Sub, SQS), the spike sits there while consumers catch up. The cost shape is also different: the buffer is cheap to scale; the compute behind it scales on actual processing rate, not on the producer's burst rate.

Step 2: Run two paths so each consumer gets the cadence it needs

Operations alerts within minutes; analytics dashboards tolerate fifteen. If both consumers read from the same processor, you're either over-spending to keep analytics on the streaming tier or under-serving operations on the batch one. A streaming consumer for ops (Flink, Spark Streaming, Kafka Streams are all viable) and a micro-batch consumer for analytics share the same buffer and run independently.

Step 3: Let elastic compute follow load instead of sitting at peak

Spike at peak, idle the rest of the day. The bill stops swinging when the worker pool grows during the spike and shrinks afterward. Combine that with the buffer above and the worst case becomes 'queue gets deep for ten minutes' instead of 'cluster falls over and on-call comes in.'

Step 4: Route invalid events somewhere other than the analytics tables

A small fraction of events are malformed every hour. They can't reach the partitioned analytics tables, that's a data-quality bug analysts will chase for a week. They also can't be silently dropped, because at this volume even a small fraction is enough rows for someone downstream to eventually notice the count is off. A separate location with the original payload and the failure reason is the third option, and the only honest one. On the canvas this is an 'error_action: dlq' on the validator.

The shape that fits

Buffer absorbs the spike; ops and analytics consume independently; events that fail validation route to a side store via a dlq error action.
  • spike_buffer (queue, Kinesis); parallelism: auto
  • ops_stream (transform, Flink); freshness: real-time; on error: dlq
  • analytics_micro (transform, Spark); freshness: < 15min; on error: dlq
  • quarantine (storage, S3)
  • analytics_lake (storage, S3); backfill: partition_overwrite
  • spike_buffer -> ops_stream
  • spike_buffer -> analytics_micro
  • ops_stream -> quarantine
  • analytics_micro -> quarantine
  • analytics_micro -> analytics_lake

Common follow-up questions