Moving Partitions

Concepts covered: Replication and leaders

Adding broker 4 to a cluster of 3 changes nothing on its own. Every existing partition keeps its replica set, broker 4 holds nothing, and only partitions created after it joined can land there. To spread the load, you move partitions, which means telling the controller a new replica set for each one and letting the new replica copy the whole partition from the leader while the topic keeps serving.

The reassignment tool

kafka-reassign-partitions.sh runs in 3 steps. --generate takes a list of topics and a --broker-list and proposes a spread, --execute submits a reassignment file, and --verify reports which partitions have finished and clears the throttles it set. The file names the target replica set per partition:

{
  "version": 1,
  "partitions": [
    {"topic": "orders", "partition": 3, "replicas": [1, 2, 4], "log_dirs": ["any", "any", "any"]},
    {"topic": "orders", "partition": 7, "replicas": [2, 4, 0], "log_dirs": ["any", "any", "any"]}
  ]
}

The first entry of replicas is the preferred leader, so the file also decides who leads once the move finishes. log_dirs can name a specific disk on the broker instead of any, which is how a partition is moved from a full disk to an empty one on the same machine.

The controller's steps during a move

For orders-3, moving from [0, 1, 2] to [1, 2, 4], the controller records broker 4 as an adding replica and broker 0 as a removing replica. The partition's replica set becomes [0, 1, 2, 4] for the duration. Broker 4 opens a fetch against the leader at the leader's log start offset and copies every segment, while the ISR stays {0, 1, 2} and writes are unaffected. When broker 4 is caught up, the leader adds it to the ISR. The controller then drops broker 0 from the replica set and the ISR, moves leadership to broker 1 because broker 0 was leading, and broker 0 deletes its copy. While broker 4 copies and sits outside the ISR, no write waits for it. Once it joins the ISR it counts like any other in-sync replica, and a write that is acknowledged only when every in-sync replica has it waits for broker 4 too.

The cost of a move

A moving replica reads the whole partition over the network and writes it to a new disk. A 400 GB partition is 400 GB of reads from the leader's disk, 400 GB across the network and 400 GB of writes, on top of the traffic the topic already carries. Unthrottled, that copy competes with the leader's followers and with the writers sending to it. The tool takes --throttle in bytes per second, which sets leader.replication.throttled.rate and follower.replication.throttled.rate on the brokers involved and lists only the moving replicas as throttled on the topic, so in-sync followers are never slowed.

The copy must also outrun the writers. A partition taking in 20 MB/s under a 50 MB/s throttle closes the gap at 30 MB/s, and a throttle below the write rate never finishes. Running --verify after the move clears the throttle settings. Skipping it leaves the throttled-replica lists on the topic and the rate settings on the brokers in place, so replication of those replicas stays capped at the old rate, and any later move that reuses them crawls at that rate too. On a topic with tiered storage, the new replica copies only the local segments and reads the remote ones from the object store when asked, which makes the move as cheap as the local retention.

Changing your mind

A move in flight can be stopped with --cancel, which removes the adding replicas and leaves the originals in place; the copied bytes are discarded. Submitting a second file for a partition already moving replaces its target. A move is also how a replication factor is raised, by naming 4 brokers where there were 3, or lowered, by naming 2.

Do

• Move a few partitions at a time and watch progress with verify
• Pick a throttle above the partition's write rate
• Run verify once every partition reports complete, to clear the throttles
• Generate a plan first and read it before executing

Avoid

• Move every partition in one file on a busy cluster
• Start a move without a throttle on a leader already near its disk limit
• Forget that the new replica copies every retained byte from the start
• Assume a new broker fills up on its own

About this lesson

This section is part of the Replication: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.