What Kafka Is For

A fan-out problem

A company's data starts as events: a customer places an order, a page loads, a payment clears, a sensor reports a temperature. Each event is produced by one system and wanted by several. A sale the order service records is wanted by 4 other systems: the warehouse that packs it, the email service that confirms it, the fraud model that scores it, and the analytics tables that count it each night.

Connecting each system that produces events to each system that wants them works for 2 systems and fails at 10. Every pair needs its own pipe, its own format, its own retry logic and its own on-call owner. 5 producing systems and 6 wanting systems make 30 pipes. Adding a 7th reader means asking 5 teams to build a pipe to it, and when the fraud model goes down for an hour, the order service either holds its events for it or drops them.

One log in the middle

Kafka puts a log between them. A log is a file that grows only at its end: a new record is written after the last one, and nothing already written is changed. The order service appends each order to the log once. The fraud model reads the log from wherever it left off, and so do the email service and the analytics job. The order service never learns how many readers exist, and a reader that was down for an hour reads the hour it missed when it comes back.

The name comes from the logs software has always kept: a database's write-ahead log, a web server's access log. Each is a file written in time order where every line is one thing that happened. Kafka takes that file, spreads it over a cluster of servers, and gives every system in the company a way to append to it and a way to read it.

Kafka's names for the roles: a producer appends records, a consumer reads them, and a broker is a server that stores the log on its disk. Brokers together form a cluster. A named log is a topic, such as orders or page_views, and a company runs hundreds of them.

What kind of system that makes it

Kafka is a storage system first. A record a producer sends is written to disk on a broker and kept there whether or not anyone has read it. The default retention is 7 days, and a topic can set its own. That property is what the fan-out problem needed: because the record is stored, a reader doesn't have to be running at the moment the writer writes. Writing and reading are decoupled in time.

A database table

• Holds the current state of each row
• An update overwrites the old value in place
• Answers a question about now: what is order 1042's status?
• Each reader runs a query, a fresh question each time

A Kafka topic

• Holds every event in the order it happened
• A new event is appended and nothing is overwritten
• Answers a question about history: what happened to order 1042, step by step?
• Each reader walks forward through the records from a position it chooses

Each row of the comparison follows from the append-only rule. Because nothing is overwritten, the log is a history, and because the history is kept, any reader can walk it. A fraud model that shipped a bug on Tuesday can be pointed back to Monday's records and run again, and the order service does nothing to help.

The rest of the lesson takes these 3 claims in turn. The next section places the store among a company's other systems, the 2 after that open it up to see how it's ordered and what each record holds, and the last one returns to the shared store and what reading it does, and does not, do to the records.

About this lesson

This section is part of the The Log: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.