Adding Partitions Remaps Keys

Concepts covered: Keys and partitioning

A topic with 3 partitions is carrying more than it should, so the operator runs 1 command: kafka-topics.sh --bootstrap-server broker1:9092 --alter --topic events --partitions 6. The command returns in under a second and nothing moves: the 3 old partitions keep every record they hold, and 3 empty partitions appear. What changes is the arithmetic for every future send. hash(key) mod 3 becomes hash(key) mod 6, and for about half of all keys that is a different number.

The pattern in the table is exact. A key stays when its hash mod 6 is 0, 1 or 2, because then mod 3 gives the same value, and it moves when the result is 3, 4 or 5. Doubling the count moves half the keys. Growing from 3 to 4 instead moves 3 keys in 4, since mod 3 and mod 4 agree only when the hash mod 12 is 0, 1 or 2. There is no partition count you can grow to that keeps every key in place, and there is no way back: Kafka refuses to reduce a partition count, because records would have to be deleted.

What a moved key looks like

Follow user-3. Before the change its 4 records sit on partition 1 at offsets 200 to 230. After the change, its next record lands on partition 4 at offset 0. Every record user-3 ever had is still in the topic, but its history is now split across 2 partitions, and the per-key order the beginner lesson promised, 1 key on 1 partition read in offset order, holds within each half and not across them. A reader that is behind on partition 1 and current on partition 4 sees the newest record first. A reader that never looks at partition 4 sees nothing new for user-3 at all.

Compacted topics make the split permanent. A compacted topic of account balances keeps the latest record per key per partition, because the cleaner works 1 partition at a time and never looks across. After user-3 moves, partition 1 keeps balance=120 as its latest and partition 4 keeps balance=95 as its latest, and both survive every compaction pass forever. A reader rebuilding a table from the topic sees 2 latest values for 1 key and keeps whichever it reads last. The clean path for a compacted topic is a new topic with the new count, refilled from the old one.

Producers find out one at a time

A producer learns the partition count from its metadata and refreshes it every metadata.max.age.ms, 5 minutes by default, or sooner when a send fails. 2 producers that write the same keys refresh on their own clocks. For up to 5 minutes after the command, producer P1 may send user-3 to partition 4 while producer P2 still sends it to partition 1, so the split interleaves both ways during the window. A transaction does not protect against this: it makes records commit together on whichever partitions the producers computed.

About this lesson

This section is part of the Producers: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.