Immutable Append-Only Logs

A log is a file that only grows. Each write goes to the end, receives the next position number, called an offset, and stays at that offset until the log is trimmed. Nothing in the middle is rewritten, and nothing is deleted one row at a time. This one rule, append only, is what makes a log cheap to write, safe to share between many readers, and possible to replay.

Offset 3 is how a log handles a mistake. The customer typed the wrong street, and in a table someone would run an UPDATE. In a log, the original order_placed stays exactly as it was, and a new address_corrected event records the fix. Both facts are true: the order was placed with one address, and the address was later corrected. A consumer that printed a label from offset 0 at 14:03 used the wrong street, and the log can show why: the correction arrived 2 minutes after the label.

• 1
• 0

Every reader keeps its own place

The log holds no record of who has read what. Each consumer stores the offset it has reached and asks for everything after it. The email service may sit at offset 5 while the analytics loader, which runs hourly, is at 2. A fraud consumer added next month starts at the oldest offset the log still keeps and reads forward. Under the 7-day retention covered below, that offset is a week old, so last year's orders come from the warehouse copy. When a consumer ships a bug, the fix is to reset its offset to the point before the bug and replay. The log did nothing special to allow this.

Reading current state from an append-only table

When the log lands in the warehouse, the table has 2 rows for order 8813 and 4 for order 8812, and a dashboard wants one row per order showing where it stands now. The answer is the latest event per entity. row_number() numbers each order's events from newest to oldest, and the row numbered 1 is the current state. Ties on event_time are broken by offset, which is the order the log received them.

WITH ranked AS (
  SELECT
    entity_id,
    event_type,
    event_time,
    row_number() OVER (
      PARTITION BY entity_id
      ORDER BY event_time DESC, log_offset DESC
    ) AS rn
  FROM order_events
)
SELECT
  entity_id,
  event_type AS current_status,
  event_time AS status_since
FROM ranked
WHERE rn = 1

Retention and compaction

A log that grows forever fills every disk, so logs are trimmed by one of 2 policies. Time-based retention drops everything older than a window, such as 7 days, which suits a log that feeds a warehouse where the full history is copied anyway. Compaction keeps only the latest event for each entity key and drops the older ones, so the log holds current state for every key but loses the path that led there. A compacted log answers 'what is order 8812 now' and cannot answer 'what was it at 14:03'.

A compacted log deletes a key by appending a tombstone, an event carrying the key and an empty payload. Even deletion is an append.

Do

• Record a correction as a new event that references the one it corrects
• Store the offset with each row when loading a log into a table
• Choose time retention when the warehouse keeps the full history
• Break event_time ties with offset, the true arrival order

Avoid

• Run UPDATE or DELETE on an event table to fix a value
• Share one offset across consumers, because the slowest stalls the rest
• Compact a log that a replay or an audit will later need
• Trust event_time alone to order events from different producers

Offset 1 records a 39.98 capture for order 8812, but the card processor's settlement shows the card was actually charged 39.89. Which change makes the records match the charge?

• Append a payment_corrected event with amount 39.89 and a reference to the capture at offset 1
• Update the payload of the payment_captured event at offset 1 so the amount reads 39.89
• Delete offset 1 and append a fresh payment_captured event carrying the corrected amount
• Append the corrected amount as a second payment_captured event with the same event_id

About this lesson

This section is part of the Event Streams lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.