Keys Pick Partitions

Concepts covered: Keys and partitioning

The hash picks the partition

Step 2 of the path, picking the partition, is the producer's call, and the default rule is arithmetic. The producer takes the key's bytes, runs them through a hash function named murmur2, forces the result positive, and takes the remainder when dividing by the topic's partition count. For trip-pings with 6 partitions the formula is hash(key) % 6, and the answer is a number from 0 to 5. The same bytes always hash to the same number, so a key always lands in the same partition for as long as the topic keeps 6 partitions. The decision runs entirely inside the producer; it could be computed with the cluster unreachable.

The hash column holds stand-in values to keep the division readable; the real murmur2 result is a 32-bit number, and the remainder step is the same.

def partition_for(key_hash, num_partitions):
    return key_hash % num_partitions


hashes = {"trip-8841": 7342601, "trip-8842": 1509223, "trip-9007": 2211984}

for key, h in hashes.items():
    print(key, "->", partition_for(h, 6))

Sharing a key buys ordering

Every record with key trip-8841 lands in partition 5, and the leader appends them in the order the producer's requests arrive. A single producer sending ping 1, ping 2 and ping 3 for that trip therefore produces offsets that increase in that order, and anyone reading partition 5 sees the trip's pings in the sequence the phone sent them. That is the whole value of a key: it names a stream whose order matters, and the hash guarantees the stream never splits across partitions. Records with different keys get no ordering guarantee, so a reader of partitions 1 and 5 may see trip 8842's ping before or after trip 8841's.

A key also lets the record be found later. Because trip-8841 can only be in partition 5, a tool looking for that trip's pings reads 1 partition out of 6 and skips the other 5.

Records without a key

A key is optional. When it is null the hash has nothing to work on, and the producer falls back to spreading: it picks a partition, sends a run of keyless records there, then moves to another partition, and over a few thousand records each partition receives roughly the same count. Any 2 keyless records from the same producer can land anywhere, so their relative order is lost the moment they are split, which is acceptable for a record nobody will ever need to line up with another. Metrics samples, log lines and events that are counted rather than sequenced are the usual null-key cases.

Keyed record

• Partition fixed by hash(key) % partitions
• All records of the key stay together, in send order
• One key can only use 1 partition's capacity
• Right for events that must line up: a trip, an order, an account

Null key

• Partition picked by the producer, changing every so often
• Records with nothing in common spread evenly
• Load uses all partitions
• Right for events counted in bulk: metrics, log lines, clicks

Overriding the default

The hash can be bypassed in 2 ways. A record can carry an explicit partition number, set when the record is built, and the producer honors it without hashing; the key still travels with the record and is stored, but plays no part in routing. Or partitioner.class can name your own class, which receives the topic, the key bytes, the value bytes and the live partition count and returns a partition number. Section 4 uses that hook. Either way, the rule that readers rely on still holds: whatever puts a key's records in one partition must keep doing so, or the order you promised downstream breaks.

Given --property parse.key=true --property key.separator=:, the console producer splits each typed line at the first colon, so trip-8841:lat=37.61,lon=-122.38 sends key trip-8841 and lands in partition 5 every time.

Changing the partition count changes the divisor, and a key that lived in partition 5 under 6 partitions may land elsewhere under 8. A later lesson works through that move; for now, the count is part of the contract.

About this lesson

This section is part of the Producers: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.