Reading Does Not Delete
Concepts covered: The append-only log
A read is a copy, and the log is the original
A reader asks a broker for records from an offset, the broker sends it a copy of the bytes, and the log is as it was. Reading changes nothing on the broker: the record is at the same offset, with the same bytes, for the next reader and for the same reader an hour later. The broker removes records on its own schedule, by age, 7 days by default on a new topic, and whether a record has been read 0 times or 50 times plays no part in that decision.
- A queue
- A worker takes a message and the message is gone
- Each message has one reader
- A second system needs a second copy of the message
- Re-reading means the producer sends it again
- A Kafka partition
- A reader copies a record and the record stays at its offset
- Each record has as many readers as show up
- A second system reads the same record at the same offset
- Re-reading means the reader moves its position back
Each reader keeps its own position
A position belongs to the reader, and moving it changes nothing in the log. A fraud scorer on the payments topic remembers it is at offset 4 of partition 0. A warehouse loader on the same partition remembers it is at offset 2, because it ran at 09:00 and read offsets 0 and 1. A new audit reader started this morning begins at 0, so the one log now has 3 positions on it, one per reader.
| Clock | Log, partition 0 | Scorer position | Loader position | Audit position |
|---|---|---|---|---|
| 09:00 | offsets 0 to 3 | 4 | 2 | not started |
| 09:30 | offsets 0 to 6 | 7 | 2 | not started |
| 10:00 | offsets 0 to 6 | 7 | 7 | 0 |
| 10:05 | offsets 0 to 8 | 9 | 7 | 9 |
Position means the next offset the reader will ask for. At 09:00 the scorer has handled offsets 0 to 3 and will ask for 4. At 10:00 the loader wakes for its hourly run, reads offsets 2 to 6 in one request and moves to 7, records the scorer finished half an hour earlier. The audit reader starts at 0 and by 10:05 has read all 9 records, 7 of which were on disk before it existed. The log column only grows, and the 3 position columns move at 3 different rates.
- 1 broker
- payments: 1 partition, replication factor 1
The scorer starts as records arrive. The loader starts 3 seconds later at the oldest record and reads the same 6 offsets. The tape is unchanged by either of them.
Starting positions
A reader that has no position yet chooses one. It can start at the oldest record the partition still holds, which is how the audit reader read history it was never running for. It can start at the end, receiving only records appended after it arrived, which is what an alerting system wants. Or it can name an offset, or hand the broker a time and start at the first record from that moment on. The choice is the reader's, and no other reader is consulted or affected.
What many readers buy you
- A system added today reads the history it missed: it starts at the oldest record still kept and catches up
- A bug in a reader is fixed by re-reading: set its position back to Monday and run again
- Readers at different speeds share one producer: milliseconds for the scorer, an hour at a time for the loader
- A reader that crashes resumes from its own position and misses nothing the log still holds
Each of these solves the fan-out problem from the first section. The producer wrote once. The scorer took what it needed on its own schedule, as did the loader and the audit reader, and none of them could damage the others, because none of them could change the log.
The one limit is the age at which the broker deletes records. A reader that falls 8 days behind on a topic kept for 7 days has lost its oldest day, and that loss has nothing to do with what any other reader did. The rule that reading does not delete describes readers only; the broker still deletes records once they pass the retention age.
About this lesson
This section is part of the The Log: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.