Hot Keys and Skew

Concepts covered: Keys and partitioning

A hot key puts all its load on 1 partition

A key buys order by pinning its records to 1 partition, and the pin holds no matter how much traffic the key carries. Suppose trip-pings were keyed by pickup zone instead of trip, so that every ping from a driver waiting at the airport carries key zone-SFO. The airport is where drivers queue, so half of all pings in the city may carry that 1 key, and all of them hash to partition 3. Partition 3's leader appends, replicates and serves half the topic's bytes while the other 5 partitions share the rest; a reader of partition 3 falls behind while readers of the others sit idle. That imbalance is skew, and a key that causes it is a hot key.

kafka-get-offsets.sh --bootstrap-server broker-0:9092 --topic trip-pings prints the end offset of every partition, which equals its record count while the log still starts at offset 0: partition 3 above holds 54% of 11.8 million records, and --time earliest shows where each log starts. kafka-log-dirs.sh --describe --bootstrap-server broker-0:9092 --topic-list trip-pings reports the bytes each partition occupies on each broker, which is the number the disks feel.

Adding partitions leaves the key hot

The reflex is to give the topic more partitions, and for a hot key it changes nothing: zone-SFO hashes to exactly 1 partition out of 12 just as it did out of 6, and that partition still carries half the topic. Partition count sets how many keys can be spread; it sets no limit on how much 1 key can concentrate. The fix has to change what the key names.

Split the key along the order you need

Ask what order the readers actually depend on. A rider's map needs 1 driver's pings in sequence; it never needs driver A's ping ordered against driver B's because both sat at the same airport. The unit whose order matters is the driver, so the key should name the driver: zone-SFO:driver-2201. Thousands of drivers mean thousands of keys, the hash spreads them over all 6 partitions, and each driver's pings still sit in 1 partition in send order. The order given up, across drivers inside a zone, was never promised to anyone. This is the general rule for a hot key: shrink the key to the finest unit that still needs ordering, and the heat spreads with it.

Keeping a coarse key with a custom partitioner

Sometimes readers key their own state on the original value and cannot take a longer key. Then the spreading moves into the producer's partitioner. A custom partitioner.class keeps the record's key as zone-SFO but routes it by a finer field taken from the value, say the driver id, hashed into a fixed set of partitions reserved for that zone. Readers still see key zone-SFO on every record; each driver still lands in 1 partition; the zone's traffic covers 2 or 3 partitions instead of 1. The partitioner must be deterministic over something stable in the record, because a partitioner that picks at random spreads the load and shreds every driver's order at the same time.

import zlib

HOT_ZONES = {"zone-SFO": [1, 3, 5]}


def choose_partition(key, driver_id, num_partitions):
    if key in HOT_ZONES:
        spread = HOT_ZONES[key]
        return spread[zlib.crc32(driver_id.encode()) % len(spread)]
    return zlib.crc32(key.encode()) % num_partitions

The function keeps the hot zone's drivers on partitions 1, 3 and 5, each driver on 1 of them for good, and sends every other key through an ordinary hash. The real class implements Kafka's Partitioner interface and hashes with murmur2 so that cold keys land where the default partitioner would put them; the Python shows the decision, which runs on the app thread at step 2 of the path for every record.

Keys that carry no order

If the zone's pings are only counted, the key was never needed: send them with a null key and let the producer spread them. Ordering across the topic is gone, and so is the hot partition.

Do

• Measure per-partition end offsets before and after a key change
• Key by the finest unit whose order readers depend on
• Make any custom partitioner a deterministic function of the record

Avoid

• Add partitions to cool a hot key; 1 key still maps to 1 partition
• Route a hot key at random; that spreads load and destroys its order
• Change a topic's key design without telling every reader of it

About this lesson

This section is part of the Producers: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.