Autoscaling and Cost Tradeoffs
Concepts covered: Pipeline Monitoring
Autoscaling sounds like the obvious answer, but interviewers probe the complexity it adds: when to scale up and how fast, when to scale down, and what over-provisioning costs. Those questions test whether you think about the business impact of technical decisions.
Autoscaling triggers
| Metric | Scale-Up Trigger | Scale-Down Trigger | Risk |
|---|---|---|---|
| Consumer lag | Lag > 100K and growing for 5 min | Lag stable near 0 for 30 min | Lag spikes from backfill can trigger false scale-up |
| CPU utilization | CPU > 80% for 3 min | CPU < 30% for 15 min | CPU may spike from GC, not actual load |
| Queue depth (SQS) | Messages > 10K visible | Messages < 100 visible | Depth includes retries and dead letters |
| Processing latency | p99 latency > SLA for 5 min | p99 < 50% of SLA for 15 min | Downstream slowness looks like consumer slowness |
Putting a cost on the scaling decision
Netflix and Stripe interviewers test FinOps awareness, and so do Airbnb's, meaning whether you can quantify the cost of your scaling decision. Most candidates never mention resource tagging or spend alerts, let alone reserved vs on-demand instances, so bringing them up reads as production maturity.
- Scale-up fast (1-2 min), scale-down slow (15-30 min). Asymmetric scaling prevents thrashing.
- Set a maximum scale to prevent runaway cost from a scaling bug or feedback loop.
- Use reserved capacity for baseline load, on-demand for spikes.
- Autoscaling Kafka consumers requires increasing partitions first. You cannot autoscale consumers beyond the partition count.
The L6 signal: 'I would tag all autoscaled resources with the pipeline name and team owner so we can attribute cost to the specific pipeline. If a pipeline's cost grows 5x month-over-month, that triggers a cost review before the next billing cycle.'
About this lesson
This section is part of the Backpressure and Scaling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.