Kappa: Stream Only, Batch Replay
Concepts covered: Kappa Architecture
One job, one log, one codebase
Kappa architecture removes the batch layer and keeps the log. Every event is written to an append-only log that retains messages for a configured window, and a single stream job reads it and maintains the output table. To change logic or undo a bug, the team starts a second copy of the job at the oldest retained position, lets it rebuild a fresh output table from history, and points consumers there once it catches up. Recomputation becomes replay: the program that handles live data handles history too.
Jay Kreps named the pattern in 2014, while at LinkedIn, in the article Questioning the Lambda Architecture, written as a response to the 2-codebase cost.
- page_view_events (source, event feed)
- event_log (queue, 30-day retention)
- count_job_v1 (transform, stream job at live offset)
- count_job_v2 (transform, same code, replaying from oldest offset)
- article_views_v1 (storage, old rule)
- article_views_v2 (storage, new rule)
- dashboard (consumer, switches v1 to v2 at cutover)
- page_view_events -> event_log
- event_log -> count_job_v1
- event_log -> count_job_v2
- count_job_v1 -> article_views_v1
- count_job_v2 -> article_views_v2
- article_views_v1 -> dashboard
- article_views_v2 -> dashboard
A replay with numbers
The news site receives 2,000 views per second on average, and its log retains 30 days, 5.18 billion events. The crawler filter ships as job v2, reading from the oldest retained offset. A replaying job never waits for events, so it runs as fast as the engine and output table allow. At 100,000 events per second it gains on the log head at a net 98,000, because new views keep arriving, and catches up after roughly 52,900 seconds, 14.7 hours. Meanwhile v1 serves the dashboard with the old rule.
| clock | job v1 | job v2 | dashboard reads |
|---|---|---|---|
| Mon 09:00 | live, old rule | started at oldest offset, 30 days behind | article_views_v1 |
| Mon 17:00 | live, old rule | 8 hours in, about 17 days replayed | article_views_v1 |
| Mon 23:41 | live, old rule | caught up, now live | article_views_v1 |
| Mon 23:45 | live, old rule | live | article_views_v2 (switched) |
| Tue 09:00 | stopped, table dropped | live, the only copy | article_views_v2 |
Lambda needed 2 edits, 2 deploys, a nightly run and a check that the views agree; Kappa needs 1 edit, 1 deploy and a table swap. The replay was also the test: v2's output over 30 days of real history sat beside v1's before any consumer switched, and diffing the 2 tables shows which articles moved and by how much.
Retention sets the replay horizon
The master dataset lived in a file archive that cost little and kept everything. The log is built for many sequential readers at once, replicates each message to several brokers so a disk loss drops nothing, and holds those bytes on the disks of brokers serving live traffic. A site writing 1TB of events a day with 3 replicas pays for 3TB of broker disk per retained day. Retention is a budget, and the replay horizon is exactly that budget: a bug found 45 days after it shipped can't be undone by replaying a 30-day log.
- 21TB
- 270TB
The plain Kappa answer keeps retention long and pays the broker bill, which works while volume is modest. Another option keeps a short hot window in the log, copies older events to cheap files, and teaches the job's reader to read the archive first and switch to the log at the archive's last offset: 1 transform, 2 input readers, and a handoff seam where each event must appear once. A team can also accept a bounded horizon and patch older history by hand. Each choice states how far back the team expects to recompute.
A Kappa pipeline retains 14 days of events. On day 60 the team finds that a deploy on day 15 has been dropping events from one app version. Which statement about recovery is correct?
- Replaying the log rebuilds days 46 through 60 correctly; days 15 through 45 stay wrong unless an archive outside the log still holds those events.
- Replaying the log from its oldest offset rebuilds all 45 affected days, because the stream job reads history and live data with exactly the same code.
- Replaying is unnecessary, because the fixed job corrects the stored counts as new events arrive and the window rolls forward.
- Replaying the log rebuilds all 45 days, as long as the replay job is started before the current 14-day window rolls past day 60.
About this lesson
This section is part of the Batch vs Streaming: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.