The Partitioned Log

Concepts covered: Offsets

A topic is several logs

A topic is split into partitions, and each partition is its own log: an ordered sequence of records that grows at its end. The orders topic created with 3 partitions is 3 such sequences, numbered 0 through 2. When a producer sends a record, the broker appends it to the end of exactly one partition, and that is where the record lives from then on. Which partition is decided by the record's key, which the next section covers; this section is about what happens once it lands.

Each row reads left to right in time. Partition 1 holds 4 records and partition 2 holds 1, which is normal: partitions fill at different rates depending on what is sent to each. The columns line up on the page and nowhere else. Offset 2 of partition 0 and offset 2 of partition 1 were written at unrelated moments, and nothing in Kafka says which came first.

Grows at the end, and only there

A partition accepts one kind of write: an append. The broker takes the record, writes it after the last record on disk, and hands back the position it got. Records are never inserted in the middle or edited after the write. They do not move, and reading one does not remove it. This is a rule about the storage, and it's what makes the rest cheap.

Set that against the work a database does for one update: find the row through an index, lock it, write the new value, update every index that points at it. An append skips all of it. That's why a broker's write speed is bounded by how fast its disk writes in sequence, and why a single broker handles hundreds of thousands of small records a second on ordinary disks.

An offset is a record's address

The position the broker hands back is the offset: a whole number, 0 for the first record ever written to the partition, and each record after it gets the next number. Offsets are assigned at append and never reused, so every record in a partition has a number no other record in that partition has or will have. Put the topic and partition together with the offset and you have a record's address, which names the same record to every broker and to every producer or consumer that uses it.

orders partition 1 offset 2 is cart-4: shipped for a reader on the fraud team and for a reader on the warehouse team. Both can say they have read through offset 2 of partition 1 and mean the same thing, and a reader that stops there tonight and asks for offset 3 tomorrow receives cart-9: placed, the record that followed.

Offsets are also how a reader keeps its place. A reader does not read a topic as a whole; it reads each partition from an offset, and after each record it has handled, its position in that partition is the offset of the next one. That position belongs to the reader, and reading never changes the log. A reader may ask the broker to store the number for it, which a later lesson covers, and the records stay as they were either way. 2 readers of the same partition hold 2 different numbers.

Order holds inside a partition

Within one partition, a record at a lower offset was appended before a record at a higher offset, and a reader receives them in that order. Across partitions there is no such promise: partition 0 offset 3 may have been written before or after partition 2 offset 0. A topic's order is the order of each of its partitions, 3 orders for 3 partitions. When the whole story of one cart has to be read in sequence, every record of that cart must land in the same partition, and the record's key is what arranges that.

About this lesson

This section is part of the The Log: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.