Mirroring to Another Cluster

Concepts covered: Replication and leaders

Replication protects a partition against a broker dying. It does nothing when the whole cluster goes: a region's power, a bad configuration push, a disk corruption that every replica copied. For that, a second cluster in another place holds a copy of the topic, written by a mirror. Apache Kafka ships one as MirrorMaker 2, started with connect-mirror-maker.sh and a properties file.

A reader on one side, a writer on the other

The mirror is a client of both clusters. It reads records from the source cluster's partitions, as any reader does, and writes each to the target cluster, as any writer does, remembering how far it has read. Because it is an ordinary client, it is asynchronous: a record acknowledged on the source is on the target only after the mirror has fetched and written it, so the target trails by the mirror's lag. It also delivers at least once: if the mirror crashes after writing a record but before saving its position, it writes that record again on restart.

Topic names and offsets on the target

Topic orders on east appears on west as east.orders, with the same partition count and each record in the same partition number it had at the source. The prefix is the source cluster's alias and does 2 jobs: a reader on west can tell local records from mirrored ones, and a mirror running the other way, west to east, skips any topic whose prefix names east, so 2 clusters can mirror each other without a record circling forever. Topic configs such as retention are copied too, so the mirror's log is cleaned by the same rules as the source.

The offsets differ. The mirror started copying after the source had deleted its early segments, so the target's first offset was paired with a much later source offset, and that difference at the start explains the 323 gap. Each duplicate the mirror wrote since then narrowed the gap by 1, since a duplicate takes a target offset of its own. Order 9004 is on east and has not reached west yet: that is the mirror's lag, and it is what a failover loses.

Failing a reader over

A reader of orders on east keeps its position as a source offset, say 1202. On west that number means nothing, since offset 1202 of east.orders is a different record, if it exists at all. The mirror solves this by writing offset syncs: a record, on a topic of its own named mm2-offset-syncs.west.internal, pairing a source offset with the target offset the same record received, written whenever the pair has moved by offset.lag.max records. That topic lives on the source cluster by default, which is useless when the source is the cluster that died, so a mirror built for failover sets offset-syncs.topic.location=target to keep it on west. To fail over, the reader finds the latest sync whose source offset is at or below its own, and starts from that sync's target offset.

A reader stopped at source offset 1202 takes sync 3 and starts at target offset 877, which is order 9001. It rereads 2 records it had already processed, 9001 and 9002, before reaching new ground at 9003. A reader at 1095 takes sync 1 and rereads 95. The reread is bounded by the sync spacing, and a reader that cannot tolerate duplicates keeps its own record of what it has processed. Then there is 9004: written to east, acknowledged, and absent from west. If east never returns, it is lost. If east returns, it sits in east's log at offset 1203, and nothing moves it to west on its own, since failing back is a second migration in the other direction.

Mirror lag is the figure to watch. The mirror reports how far behind each partition it is, in records, and at a source taking 10,000 records a second, a lag of 2 seconds is the count of acknowledged records a failover at that moment drops.

• 20,000

Mirroring through an unclean election

The mirror takes the source's offsets at their word, so a source partition that reuses offsets, as an unclean election does, damages the copy in 3 ways. Say the mirror had copied source offsets up to 89 when a stale replica took over leading from offset 61. First, the target keeps the old records 61 to 89: the mirror never deletes what it has written, so west holds records east has lost for good. Second, the mirror's read position now sits among reused offsets, and where it resumes decides what follows: from the divergence point, the new records 61 onward are copied after the old ones and the target holds both versions; from its old position, the new records below 90 never reach the target. Third, the syncs written before the rewind still pair source offsets 61 to 89 with target offsets of the lost records, and syncs written after it pair the same source numbers with new target offsets, so a source position in that range can translate to the wrong target record. Nothing on either cluster flags this, which is a reason to keep unclean election off on any topic mirrored for failover.

Readers that keep their positions on the cluster itself can have the mirror translate them in bulk; that mechanism belongs to a later lesson on consumer groups.

About this lesson

This section is part of the Replication: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.