Horizontal Scaling With Partitions

Concepts covered: Partitioning Strategies

Horizontal scaling in Kafka means adding partitions and consumers together. Each partition is consumed by exactly one consumer in a consumer group. Consumers beyond the partition count sit idle, and partitions without consumers waste Kafka storage, so the 2 have to scale in lockstep.

Horizontal scaling: partitions and consumers scale together (one partition per consumer). Adding consumers beyond the 8 partitions does nothing; the slowest link in the chain caps throughput.

producers (source)
kafka_topic (queue, Kafka, 8 partitions)
consumer_group (transform, Flink); parallelism: 8 consumers
warehouse (storage, Snowflake)

producers -> kafka_topic
kafka_topic -> consumer_group
consumer_group -> warehouse

How partitions map to consumers

Partitions	Consumers	Parallelism	Notes
6	3	3x	Each consumer handles 2 partitions. Under-provisioned consumers.
6	6	6x	Optimal. One consumer per partition. Maximum parallelism.
6	9	6x	3 consumers sit idle. Wasted resources. Cannot exceed partition count.
12	6	6x	Each consumer handles 2 partitions. Can add 6 more consumers without repartitioning.

A strong-hire candidate says something like 'I would set the initial partition count to 2-3x the current consumer count so we can scale consumers without repartitioning. Increasing partitions is easy; decreasing them requires topic recreation.'

Partition key design

The partition key determines which partition a record lands in, and a bad key creates hot partitions. If you partition by customer_id and one customer generates 50% of events, one consumer handles 50% of the load while the others idle. The interview tests whether you think about data skew.

Do

Choose a high-cardinality key (user_id, device_id) that distributes evenly
Over-provision partitions so you can scale consumers later
Monitor per-partition lag, not just aggregate lag

Avoid

Use a low-cardinality key (country, status) that creates hot partitions
Add consumers beyond the partition count (they sit idle)
Assume uniform distribution without checking actual key distribution

The follow-up to expect

Follow-Up: This is the data skew trap. Strong answer: 'I'd add a random suffix to the key for the hot entity: customer_123_shard_0 through customer_123_shard_9. This spreads one customer's events across 10 partitions. The consumer reassembles them by stripping the suffix.' This is called salting.
Follow-Up: Ordering requires all messages for a customer to go to the same partition. Strong answer: 'Partition by customer_id for ordering. If that creates skew, I'd accept the skew and scale the consumer for that partition vertically, or rethink whether strict ordering is actually required for all customers.'

About this lesson

This section is part of the Backpressure and Scaling: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.