Reading the Nearest Replica

Concepts covered: Replication and leaders

By default every read of a partition goes to its leader, wherever it sits. With replicas spread across 3 zones, a leader in zone a serves readers in zones b and c across zone boundaries, which cloud providers bill per gigabyte and which adds milliseconds to each fetch. The replicas in b and c hold the same committed records and do nothing with them but wait. Since KIP-392, a reader can fetch from the follower in its own zone instead.

Broker and reader settings

The broker needs replica.selector.class set to the rack-aware selector and a broker.rack naming its zone, which rack-aware placement already requires. The reader sets client.rack to the zone it runs in. On the reader's first fetch, the leader answers with no records and a preferred read replica, the follower whose rack matches; the reader sends its next fetches there.

broker.rack=us-east-1b
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector

client.rack=us-east-1b

How far behind a follower reads

A follower serves only up to its own high watermark, and it learns the leader's high watermark one fetch late: the leader advances it when every in-sync replica has fetched a record, and tells the followers in the response to their next fetch. A record committed at the leader at 10:00:00.000 becomes readable from the leader at once and from the follower after the follower's next round trip, which is bounded by replica.fetch.wait.max.ms, 500 ms by default, plus the network.

Two readers of the same partition in different zones therefore see each record at different moments, around 140 ms apart in this table and up to half a second in a quiet partition. They never see different records at an offset, and never a record the leader has not committed, because the follower's high watermark never passes the leader's. The order inside the partition is unchanged. What changes is that latest means something different for each reader: the end of the log as the follower knows it.

Falling back to the leader

The leader picks the follower only from the ISR, so a reader is never pointed at a replica that fell behind. If the chosen follower then drops out of the ISR or dies, the reader's fetch fails, it goes back to the leader, and the leader names a new preferred replica or serves the reader itself. Every metadata.max.age.ms, 5 minutes by default, the reader asks the leader again, which lets the leader move it after a reassignment puts a replica in its zone. If no replica shares the reader's rack, it reads from the leader as before. Writes are untouched: a follower never accepts a record, so the reader's zone decides where its reads go and nothing about where writes go.

About this lesson

This section is part of the Replication: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.