Choosing a Partition Count

The partition count is set when a topic is created, and it decides order, parallel reading, placement and cleanup. Records are ordered within one partition and not across them. A partition is read in order by 1 reader at a time, so 8 partitions let 8 readers work at once while a 9th waits idle. Each partition is a directory on 1 broker, so the count decides how a topic's bytes and writes spread across the cluster. Expiry and compaction both work partition by partition too.

Spreading across brokers

A 6-partition topic on a 3-broker cluster puts 2 directories on each broker, so its disk and its writes split 3 ways. A 2-partition topic on the same cluster leaves 1 broker holding none of it. The count therefore also caps how many brokers a topic can use, which matters most for the hot topic whose disk and network a single broker could not carry.

A key's partition depends on the count

A record with a key goes to the partition at hash(key) mod partition count, and the Java client computes that hash with murmur2. With 3 partitions, cust-7 maps to the same partition every time, which is what lets a compacted partition hold the key's latest value and lets a reader see the key's records in order. Change the count to 4 and hash mod 4 sends cust-7 somewhere else. The animation sends the same 8 records, 4 keys, to 2 topics side by side: orders with 3 partitions and orders-v2 with 4, or with 6 at the switch.

Follow 1 key across the 2 topics. In orders its 2 records land together, and in orders-v2 they land together again, though some keys sit on a different partition number than they do in orders. Either way a key's records stay together; only the directory that holds them changes. If orders were a topic's past and orders-v2 its future at a higher count, a key that moved would be split across 2 directories.

Adding partitions moves nothing

kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic orders --partitions 4 creates 1 new empty directory on a broker and tells producers that there are now 4 targets. No existing record moves, because the segments are closed files and compaction and retention are the only things that rewrite them. A key's history now lives in 2 places: its past in the partition it used to hash to, its future in the partition it hashes to now. For that key, 2 things break. Its order across the change is gone, since a reader working 2 partitions can be anywhere in one when it reads the other. And on a compacted topic, the old partition keeps the key's old value forever; a tombstone sent later goes to the new partition and removes nothing in the old one, so a reader rebuilding a table from offset 0 can finish with the stale value, depending on which partition it reads last.

The count can only go up. There is no command to merge partitions, because the records in the removed partitions would need new offsets, and offsets are fixed at write time. A topic that needs fewer partitions, or a compacted topic that needs more, is a new topic: create it with the right count, copy the records across, move the readers, then drop the old one.

Picking the number

Start from throughput. Measure what 1 reader can process per second and what 1 partition can take, then divide the target rate by the smaller of the 2: 30 MB/s of writes with readers that handle 4 MB/s each needs 8 partitions to keep up, and 12 leaves room for growth without a change later. Then weigh what each partition costs. Every one is a directory with its own segments and 3 files per segment, so open files, index memory and the retention thread's work all scale with the count times the segments per partition. A topic of 1,000 partitions, each holding a week of 1 GiB segments rolled once a day, keeps about 7,000 segments and over 21,000 files on the cluster.

• 1
• 50

Do

• Measure one partition's throughput and divide the target by it.
• Create with room to grow; 12 partitions on a small topic cost little.
• Treat a compacted topic's count as fixed; grow it with a new topic.

Avoid

• Add partitions to a topic whose readers depend on per-key order.
• Create 1,000 partitions to be safe; each is a directory of files on a broker.
• Plan on shrinking a topic; the count only goes up.

About this lesson

This section is part of the The Log: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.