KRaft Controllers

Concepts covered: Replication and leaders

Every election in the last section was carried out by a controller: the one part of the cluster that holds the list of every partition's replicas, leader and ISR, and that writes down a new leader when a broker dies. In Apache Kafka 4.4 that controller runs as a small quorum of its own, using the Raft protocol, and keeps its metadata in a log that works like any partition you have read so far. The name for this is KRaft.

The roles a node can declare

Each node declares process.roles: controller, broker, or broker,controller for both on one process. Controllers form the quorum; brokers hold topic partitions. Running both on 1 process suits a laptop, and a production cluster keeps them on separate machines so that a broker drowning in fetches never delays an election.

process.roles=controller
node.id=1
listeners=CONTROLLER://ctl-1:9093
controller.listener.names=CONTROLLER
controller.quorum.bootstrap.servers=ctl-1:9093,ctl-2:9093,ctl-3:9093
log.dirs=/var/lib/kafka/metadata

A broker's file names the same bootstrap list under process.roles=broker, which is how it finds the quorum. Each node is formatted once with kafka-storage.sh format, which stamps the cluster id into its log directory, and a controller formatted with --initial-controllers joins as a voter from the start.

The metadata log

The quorum's state is a topic named __cluster_metadata with 1 partition, held by every controller. One controller is its leader, called the active controller, and the others replicate it as followers, with the same fetch loop a topic partition uses. A record in it counts as committed once a majority of the voters have written it: 2 of 3, or 3 of 5. Each record describes one change: a topic created, a partition's ISR shrinking, a broker registering, a leader changing. Replaying the log from the start rebuilds the whole cluster's metadata, which is what a controller does when it boots, and it snapshots the log periodically so that the replay stays short.

Fencing a dead broker

Each broker sends a heartbeat to the active controller on a fixed interval, and when the controller has heard nothing for longer than the session timeout it fences the broker: it writes a record marking it fenced, then one PartitionChangeRecord per partition that broker led, naming a new leader from each ISR. Those records commit once a majority of controllers hold them. Each broker learns by fetching: it runs a fetch against the metadata log, exactly as a follower fetches a partition, and applies every new record it reads. A broker finds out it has become leader of orders-3 by reading that fact from the log.

• 2 s
• 9 s

Watching the quorum

kafka-metadata-quorum.sh --bootstrap-server b1:9092 describe --status prints the active controller and how far each voter has replicated:

Brokers appear as observers: they replicate the log without a vote. A standby whose lag climbs is one that would lose records if it won an election, and the Raft rule refuses to elect it, since a candidate wins only with votes from a majority whose logs are no longer than its own. That rule is why the controller never has an unclean election of its own. Losing a minority of controllers costs nothing. Losing a majority stops the cluster from changing metadata until enough return: existing leaders keep serving reads and writes, and every election, topic creation and ISR change waits.

About this lesson

This section is part of the Replication: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.