Horizontal Scaling: Partitions and Parallelism

Horizontal scaling in Kafka means adding partitions and consumers together. Each partition is consumed by exactly one consumer in a consumer group. Adding consumers beyond the partition count wastes resources. Adding partitions without consumers wastes Kafka storage. They must scale in lockstep. The Partition-Consumer Relationship The strong-hire move: 'I would set the initial partition count to 2-3x the current consumer count so we can scale consumers without repartitioning. Increasing partitions is easy; decreasing them requires topic recreation.' Partition Key Design The Follow-Up Trap