Tombstones and Deletes
Compaction keeps the latest record for every key forever, so a customer who closes their account still has a current address in the topic a year later. There is no command that removes a key from a partition. The producer removes it by writing one more record: the same key with a null value, called a tombstone. To the cleaner the tombstone is the key's latest record, so the older values for that key are dropped on the next pass. To a reader it is an instruction: delete this key from whatever you are building.
producer.send("customer-addresses", key=b"c-7", value=None)
producer.flush()The broker stores a tombstone like any other record. It has an offset, a timestamp and a key, and it sits in the active segment until that segment rolls. A reader fetching it receives a record whose value is null, which differs from a record whose value is an empty byte array; only the null form is a tombstone to the cleaner, and an empty value is kept as the key's latest value like any other.
Why the tombstone must linger
Suppose the cleaner removed the tombstone in the same pass that removed the old values. A reader that was halfway through the partition, or offline for the night, would reach the end and never learn that c-7 was deleted; its table would carry a stale address with nothing left in the topic to correct it. So the tombstone is kept after the values are gone, for delete.retention.ms, 24 hours by default, measured from the moment the cleaner first processed the segment that holds it. A later pass after that time removes it. A reader that reads through within the window sees the tombstone and deletes the key. A reader that arrives after it sees no record for c-7 at all, and that is also correct: the key is absent.
| moment | what happens to c-7 | what a reader starting at 0 ends with for c-7 |
|---|---|---|
| Mon 10:00 | tombstone written at offset 812, in the active segment | Oslo at 0, Bergen at 2, Tromso at 5, then the tombstone: no c-7 |
| Tue 02:00 | the segment holding 812 rolls and closes | no c-7 |
| Tue 03:00 | the cleaner rewrites that segment: offsets 0, 2 and 5 are dropped, 812 is kept and stamped with Tue 03:00 | only the tombstone: no c-7 |
| Wed 03:05 | a pass finds 812 older than delete.retention.ms and drops it | nothing for c-7: no c-7 |
Each row ends in the same state for a reader that starts from offset 0, which is the point of the design. The window matters for a reader that already holds c-7 from an earlier read and comes back later: if it returns before Wednesday 03:05 it sees the tombstone and deletes the key; if it returns after, nothing in the topic mentions c-7, and the stale row survives in its table.
What a reader does with each record
| record | reader's action on its table |
|---|---|
| key with a value | insert the key, or overwrite the value it already holds |
| key with a null value | delete the key |
| a key seen twice in the tail | apply both in offset order; the later one is the state |
A reader that skips null values, which some deserializers tempt it to do by failing on an empty payload, keeps every deleted key. Processing in offset order makes the rest work without special cases: in the Monday row above the tombstone sits in the active segment behind 3 live values the cleaner has not touched, and a reader applying each record in turn still finishes with c-7 removed.
What a tombstone does not do
About this lesson
This section is part of the The Log: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.