How Many Kafka Partitions? Count, Consumers & Sizing

The 2 rates are measurements, not constants, and the example values (10 MB/s and 5 MB/s) are assumptions to replace with your own. Measure the producer rate against a topic with 1 partition, using your real record size and your batching and compression settings. Measure the consumer rate with the real processing code, because the slowest step usually sits after the read: a database write or an API call, not Kafka.

In the example the consumer side decides: 12 partitions would carry the writes, but consumers at 5 MB/s per partition need 24. When the consumer side is the larger number, the cheaper fix is often a faster consumer, since halving the processing cost per record halves the partitions needed. Take the count from the faster consumer only if you've measured that it exists.

Plan for the peak you expect over the life of the topic, not today's average. Since the count can be raised later at a cost to key ordering, a modest margin of 1.5x to 2x on the computed number is a reasonable place to start, and a number that is several times larger buys nothing but overhead.

Partitions are handed out whole, so a count that does not divide by the group size leaves some consumers with more. With 24 partitions, a group of 6 consumers gives each 4 partitions and a group of 8 gives each 3. A group of 10 gives some consumers 3 and others 2, so its busiest consumer carries the same 3 as a group of 8 and the 2 extra consumers buy no relief.

The busiest consumer sets the group's pace, because it falls behind first. A count with many divisors, such as 24, lets the group grow through 2, 3, 4, 6, 8 and 12 consumers with an even split each time. A prime count like 13 or 23 splits evenly only at 1 consumer or at 1 per partition. Idle consumers appear when a group is larger than the count: 4 idle at 28 consumers on 24 partitions.

Kafka guarantees that a consumer reads a topic-partition's events in the order they were written, and that events with the same key go to the same partition. That gives per-key ordering, for example every event for 1 customer in sequence, and none across keys. If a topic needs 1 global order, it needs 1 partition and at most 1 active consumer per group, which caps its throughput at what 1 partition carries.

The key-to-partition mapping depends on the count. The Apache Kafka operations documentation warns that with the default partitioner, which uses a hash of the key modulo the number of partitions, raising the count changes the mapping, so messages with the same key may route to different partitions after the expansion and per-key ordering can break for existing keys. Kafka doesn't redistribute existing data.

The same page states that Kafka does not currently support reducing the number of partitions for a topic. Together these make the count a one-way decision for any topic whose consumers depend on key order or keep per-key state: pick it with growth in mind, or plan a new topic and a migration. A topic whose consumers keep no per-key state can grow with little trouble.

A producer keeps a batch per partition it writes to, sized by batch.size (16,384 bytes by default), all drawn from buffer.memory (33,554,432 bytes by default), enough for 2,048 full batches. A consumer can fetch up to max.partition.fetch.bytes per assigned partition, 1 MiB by default, so its memory use grows with the partitions it owns. Brokers keep files and replicas for each partition.

Spreading the same traffic over more partitions also makes each batch smaller, since a producer fills 1 batch per partition and sends it when it's full or when linger.ms expires. Smaller batches mean more requests for the same bytes. A topic with far more partitions than the throughput needs pays these costs for nothing, so choose the smallest count that meets the sizing and consumer targets with some margin.

Interviewers ask this as a design question: pick a partition count for an event stream and defend it. A strong answer gives the throughput arithmetic from both sides, names the key that preserves order, states the consumer group ceiling and says how to change the count later and what that breaks. It works as a short calculation to practice, so run the sizing formula with your own numbers.

Kafka questions show up across the Kafka interview questions set and in design rounds. The data pipeline interview questions and data pipeline practice problems cover the streaming pipelines these partitions feed, and the data engineer interview prep guide maps the rest of the loop. For the basics of topics, producers and brokers, see the Kafka tutorial.