Where Kafka Fits
The path an order takes
Trace one order at an online store. A customer pays, and the checkout service writes the order to its own database, which keeps the current state the storefront needs: the order exists, it is paid, it hasn't shipped. In the same request the checkout service produces a record to the orders topic. From that moment the order is in 2 places, and the 2 places serve different readers. The database serves the storefront's next page load. The topic serves everyone else.
- Checkout service (source, produces to orders)
- Storefront (source, produces to page_views)
- Checkout database (storage, current state)
- Kafka (queue, Apache Kafka)
- Fraud scorer (transform, reads each record)
- Search index (consumer, reads continuously)
- Warehouse loader (consumer, reads hourly)
- Alerting (consumer, reads continuously)
- Checkout service -> Checkout database
- Checkout service -> Kafka
- Storefront -> Kafka
- Kafka -> Fraud scorer
- Kafka -> Search index
- Kafka -> Warehouse loader
- Kafka -> Alerting
- Request path: Checkout service, Storefront, Checkout database
- The log: Kafka
- Readers: Fraud scorer, Search index, Warehouse loader, Alerting
Kafka sits between the systems that record what is happening and the systems that act on it or analyze it. On the left are services that answer user requests in milliseconds, so they write the event and move on. On the right are systems with their own clocks: a fraud scorer that wants each order within a second, a search index that updates within a minute, a warehouse load that runs every hour, and a report that runs once a night. The log lets each side keep its own clock.
What the writing side relies on
- A fast write: the broker takes the record, acknowledges it, and the checkout service returns to the customer
- A single destination: one topic to write to, however many systems read it
- Durability: once acknowledged, the record is on a broker's disk and stays for days
- No knowledge of readers: adding or removing a reader changes nothing in the checkout service
The fraud team changes what the scorer consumes by changing the scorer. It reads orders, and nothing in checkout moves. When the scorer is replaced next year by a different system, the replacement reads orders too, from whatever position its team chooses, and the checkout service isn't told.
What the reading side relies on
- Order: records within a partition arrive in the order they were appended
- Its own position: each reader decides where it starts and keeps its own place
- Independence: a slow or crashed reader costs the other readers nothing
- Replay: a reader can go back over records it already read
The warehouse loader that runs hourly reads an hour of records in one pass. The alerting system reads every record as it arrives, so it sits at the end of the log and waits for the next one. Both read the same topic and the same records, and the loader's 59 minutes of absence cost the alerting system nothing. A third team can start a reader tomorrow without a meeting with either of them. How each reader keeps its place is the subject of this lesson's last section.
Some systems stand on both sides. The fraud scorer reads orders and produces to a fraud_scores topic that the alerting system and the warehouse loader read in turn. A stream of events is often several topics long, each one a producer's output and a consumer's input, and the cluster holds all of them.
A producer or consumer is given the addresses of a few brokers and learns the rest of the cluster from them, so it names a topic and never needs to know which broker holds it.
Where the data goes next
Kafka keeps days of recent history and serves it to readers in order. A question such as how many orders region West placed last quarter belongs to the warehouse, which a reader fills from the topic and keeps for years. The storefront's question, whether this order has shipped, belongs to the database the checkout service already writes. Kafka is the hand-off between the 2: everything that happens passes through it, and each system on the right takes what it needs and stores it in the shape it needs.
| System | What it holds | Who asks it |
|---|---|---|
| Checkout database | The current state of each order | The storefront, on every page load |
| orders topic | Every order event, in order, for 7 days | Fraud scorer, search index, loaders, alerting |
| Data warehouse | Every order ever, as tables | Analysts and the nightly report |
Nothing in this section ran a query against Kafka. Readers walk the log forward; questions about totals and joins are asked of the systems the readers fill.
About this lesson
This section is part of the The Log: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.