Event Streams

In a request-driven design, the checkout service finishes an order by calling the inventory service, then the email service, then the analytics service, and waits for each reply. Adding a fraud check means changing checkout again. When the email service is down, the order either fails or checkout grows a retry branch for that one call. In an event-driven design, checkout writes one order_placed event to the log and returns. Inventory, email, analytics and fraud each subscribe to the log and react when they are ready. Checkout holds no list of who reads its events, so a new consumer costs the producer nothing.

The log sits between producer and consumers, which is why these systems are called decoupled: the 2 sides share a contract about the event, and nothing else.

Every event carries an envelope, the fields that are the same for every event type, and a payload, the fields specific to this type. The envelope is what routing, deduplication and loading rely on, so it is strict. The payload is what the business reads, so it changes as the business does, and a schema_version in the envelope records which shape the payload has.

Offset 3 is how a log handles a mistake. The customer typed the wrong street, and in a table someone would run an UPDATE. In a log, the original order_placed stays exactly as it was, and a new address_corrected event records the fix. Both facts are true: the order was placed with one address, and the address was later corrected. A consumer that printed a label from offset 0 at 14:03 used the wrong street, and the log can show why: the correction arrived 2 minutes after the label.

When the log lands in the warehouse, the table has 2 rows for order 8813 and 4 for order 8812, and a dashboard wants one row per order showing where it stands now. The answer is the latest event per entity. row_number() numbers each order's events from newest to oldest, and the row numbered 1 is the current state. Ties on event_time are broken by offset, which is the order the log received them.

A compacted log deletes a key by appending a tombstone, an event carrying the key and an empty payload. Even deletion is an append.

Most applications store state and discard how it got there. An accounts table says the balance is 240.00, and nobody can say which deposits and withdrawals produced that number. Event sourcing turns the storage inside out: the events are the source of truth, and the balance is a result computed from them, as many times as anyone needs.

The last column is computed, never stored: start at 0 and apply each event in order, adding on deposited and subtracting on withdrawn. Any state that can be described as a start value plus a sequence of changes can be rebuilt this way, which covers account balances, inventory levels, order status, and a document's edit history.

The WHERE clause is the part a state-based table cannot offer. Change the timestamp to March 12 and the same query returns 300.00, the balance as it stood that day, with no history table and no archived copy. Audits ask that question constantly. The second gift is repair: if the code that computes balances had a bug that subtracted each withdrawal twice for 3 weeks, the events are still right, because they record what was requested. Fix the code, run the fold again, and every balance is correct without a single hand-written correction.

Dashboards do not run this fold on every page load. A pipeline runs it on a schedule and writes the result to a plain account_balances table, called a read model. A read model is derived, so it can be dropped and rebuilt whenever the code or the question changes.

Current balance: 300.00 from the snapshot, minus 60.00 from offset 13, with no replay of offsets 10 to 12.

Two identity columns matter. anonymous_id is a random id the app assigns before anyone logs in, and user_id is filled once they do. A shopper browses 20 products anonymously, logs in to buy, and the login event, which carries both ids, is the only thing that connects the browsing to the purchase. Drop either column and every funnel breaks at the login step.

The event name is the vocabulary everyone shares, so it stays small and follows one pattern: object, then past-tense action. What varies goes in properties. product_viewed with product_id 220 is one event name that covers the whole catalog, while product_220_viewed is a new event name per product and a dashboard nobody can maintain. The tracking plan is the table that says which names exist, when each fires, and which properties are required.

The app records taps. A visit is derived: a run of events from one anonymous_id with no gap of 30 minutes or more between neighbours. lag() fetches the previous event's time for the same person, a gap of 30 minutes or a missing previous event flags a new session, and a running sum of those flags numbers each session. The cutoff is a convention, and a video app measuring binge watching might pick 4 hours.

A funnel asks how many sessions reached each step. Counting events gives the wrong answer, because one session can view 12 products and add 3 to the cart. The query reads sessionized_events, the October 20 output of the sessionization query saved as a table, where one anonymous_id with one session_number is one visit. The inner query collapses each visit to 3 flags, and the outer one counts the sessions with each flag set.

Every event has 2 times. event_time is when it happened, stamped by the producer, often a phone. ingested_at is when the log accepted it. For most events the 2 are seconds apart. For a phone in a tunnel, a laptop closed mid-upload, or an app that batches events and sends them every 15 minutes, event_time can be hours behind ingested_at. A tip tapped at 08:35 while the phone was offline keeps that event_time and reaches the log at 12:40, when the phone reconnects. The event is late, and it is still true.

Row 3 is the subtle case. The page view happened on October 20 and arrived on October 21. If the loader files events by arrival day, it lands in the wrong partition and October 20 is undercounted forever. File it by event_time, into event_date 2026-10-20, and a partition for a day already closed changes. A daily aggregate built at 01:00 on October 21 counted 1,204,118 events for October 20. By October 23 the true count was 1,209,744, about 0.5% higher. Neither figure is a bug. The aggregate was built before the data finished arriving.

The phone that failed to upload evt_a2 at 08:35 retries at 12:40 and again at 12:41, and the log holds 2 copies. The retry carries the same event_id, which is the reason the envelope has one. Dedupe on event_id inside each event_date partition, keeping the earliest ingested_at, with the same row_number() pattern that found the latest event per order.