One Key, Two Partitioners

Concepts covered: Keys and partitioning

The rule from the beginner lesson was: a keyed record goes to hash(key) mod partitions. That rule does not say which hash. The Java client's built-in partitioner runs murmur2 over the key's serialized bytes, clears the sign bit with & 0x7fffffff, and takes the remainder by the partition count. librdkafka, the C library underneath the Python, Go, .NET and Node clients, ships with partitioner=consistent_random, which runs CRC32 over the same bytes and takes the remainder. Both functions are deterministic and spread keys evenly, and they are different functions, so a topic written by a Java service and a Python service with the same keys ends up with each key in 2 partitions, 1 per library.

Take user-1. The Java client computes murmur2 of the 6 bytes of user-1 and gets -1118138342; clearing the sign bit gives 1029345306, which divides by 6 exactly, so partition 0. librdkafka computes CRC32 of the same 6 bytes and gets 1393466234, remainder 2, so partition 2. The 2 results agree for a key with probability 1 in 6, and none of the 6 keys above agrees. Each library is internally consistent: every user-1 record from the Java service lands on 0, every one from the Python service on 2, and a reader of partition 0 sees half of user-1's history in order with no sign that the other half exists.

The bytes matter as much as the function. Both libraries hash the serialized key, so 2 producers using the same library still disagree if 1 serializes the user id as the string 42 (2 bytes) and the other as a 4-byte integer. A topic's partitioning contract is 3 things together: the key's bytes, the hash function and the partition count, and any producer that changes 1 of the 3 is writing to a different place.

Making them agree

partitioner=murmur2_random

librdkafka's murmur2_random reproduces the Java client's function bit for bit, including the sign-bit mask and the random spread for records with no key, so a Python producer with that 1 line puts user-1 on partition 0 next to the Java service. The reverse is possible too: partitioner.class on the Java side accepts a class that computes CRC32, which someone has to write. The decision is per producer, so a mixed fleet needs every producer configured the same way, settled before the first record, because every record already written stays where it landed.

Do

• Write the hash and the key serializer into the topic's contract, next to the schema.
• Set partitioner=murmur2_random on every librdkafka producer of a shared topic.
• Test a sample of real keys through each client and compare the partition numbers.

Avoid

• Assume the same key means the same partition across languages.
• Fix a mismatch by switching 1 service while the other keeps its old history.
• Hash a key that 1 service serializes as text and another as a number.

The _random suffix names what the partitioner does with a record whose key is null: consistent_random and murmur2_random pick a random partition for keyless records, while plain consistent and murmur2 hash the empty key and send every keyless record to 1 partition. The Java client sticks keyless records to 1 partition until a batch fills or lingers out, then moves on. None of this touches keyed records, so a mismatch between libraries shows up only where order matters.

About this lesson

This section is part of the Producers: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.