Doubling Every Six Months
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L7
Problem
Our data volume doubles every six months, and each server we add makes costs climb faster while 10x flash-sale spikes still take hours to recover from. Design a pipeline that scales automatically without manual provisioning for the teams and enterprise customers who read this data.
Tuesdays are quiet. Black Friday is not.
Business requirements
- The CTO wants cost per unit of data to go down as we grow, not up.
- Operations needs to see issues as they happen, analytics works hourly, and data science only runs once a week.
- Flash sales drive huge spikes that the current system can't keep up with, and recovery has been taking hours.
- Three new enterprise customers signed contracts requiring their data be kept separate from everyone else's.
Worked solution and explanation
What this really is
This is a cost-curve problem dressed up as a scaling problem. The real question is whether you can separate storage from compute so that idle costs nothing, and then hang three serving paths off one cheap, durable layer. Anyone can draw Kafka into Flink. The trap is treating 'scales automatically' as 'a bigger cluster with autoscaling turned on'. Do that and the bill still rises with volume, a flash sale still saturates compute, and the enterprise audit fails because every tenant sits in one table.
Most candidates start by keeping the fixed cluster, adding worker autoscaling, and filtering tenants with WHERE customer_id = ? in the BI tool. The cluster idles at its floor most of the week. Every query scans one giant partition. Isolation then depends on every analyst remembering a filter. Each of the four constraints breaks at a different seam, and none of them is fixed by adding servers.
Walk the requirements
Step 1: Anchor on partitioned object storage
Cost per unit only falls when you pay for bytes at rest and for compute only while it runs. Land everything in object storage, partitioned by date and tenant. A 24/7 cluster bills for peak capacity every hour of the year, and that bill is exactly what has been exploding.
Step 2: Give each cadence its own path
Ops needs sub-minute, so it gets a narrow Flink stream into Grafana. Analytics is hourly, so an on-demand SQL engine reads the lake for the analytics dashboard. Data science is weekly, so a Spark job reads the same lake, spins up once a week, and shuts down. If you skip the weekly path, data science ends up competing with hourly reporting for the same compute.
Step 3: Absorb spikes in a buffer, then scale behind it
Kafka holds a 10x flash sale while consumers catch up. Buffering only fixes half the problem, though. The stream processor behind it carries parallelism: auto, so it adds slots during the spike and releases them afterward. Recovery time then becomes the time to drain the buffer, not hours of a failing cluster.
Step 4: Put tenant isolation in the layout
Each enterprise customer gets its own prefix in tenant_partitioned_lake, protected by storage permissions. Any query that has not been granted that prefix cannot read it. A WHERE clause is a convention, but a prefix is an actual boundary.
The reference design
- ingest_buffer (queue, Kafka)
- stream_processor (transform, Flink); freshness: real-time; parallelism: auto
- ops_dashboard (consumer, Grafana)
- tenant_partitioned_lake (storage, object storage)
- on_demand_query (transform, SQL); freshness: < 1h
- analytics_dashboard (consumer, dashboard)
- weekly_batch_job (transform, Spark); parallelism: auto
- data_science_notebooks (consumer, Jupyter)
- ingest_buffer -> stream_processor
- ingest_buffer -> tenant_partitioned_lake
- stream_processor -> ops_dashboard
- tenant_partitioned_lake -> on_demand_query
- on_demand_query -> analytics_dashboard
- tenant_partitioned_lake -> weekly_batch_job
- weekly_batch_job -> data_science_notebooks
Common follow-up questions
- When a flash sale doubles traffic for an hour, what saturates first and what catches up first? (Where back-pressure lives: buffer depth, processor scale-out speed, query cold-start.)
- How do you cap one enterprise customer's query cost without rewriting the pipeline? (Per-tenant credentials or chargeback built on the prefix layout you already have.)
- Data science wants daily instead of weekly. What changes? (Only the Spark schedule changes, because the lake already holds the data.)