Lambda to Kappa Worked Example

Concepts covered: Kappa Architecture

The workload

A podcast platform serves listens per episode to creators and its recommendation ranking, from 3,000 listen events per second. Under Lambda, a nightly recompute of episode_listens_batch starts at 02:00 over a file archive of every listen and lands roughly 07:00; a speed layer keeps episode_listens_rt since midnight; consumers read both through a merge query. The codebases drifted twice in a year, and the layer holding the stricter rule set which way totals moved at 07:00. A crawler rule added to batch only made totals drop at the landing, then climb too fast all day. A completed-listen rule raised from 30 to 60 seconds played, in the speed layer only, made totals jump at the landing, then grow too slowly.

One definition of a completed listen

The code diff, by intent

• Deleted: The batch implementation and read-time merge go; both consumers read episode_listens_v3.
• Promoted: The stream job holds the only definition of a completed listen.
• Added: The job writes episode_listens_vN; each replay is a new N and never overwrites the table being read.
• Added: Each count change emits a replacement row, and the sink overwrites by key.
• Added: Run before every cutover, it lists episodes whose counts differ between old and new tables.

SELECT
    COALESCE(o.episode_id, n.episode_id) AS episode_id,
    o.listens AS old_listens,
    n.listens AS new_listens
FROM episode_listens_v2 o
FULL OUTER JOIN episode_listens_v3 n
    ON n.episode_id = o.episode_id
WHERE o.listens IS NULL
    OR n.listens IS NULL
    OR o.listens <> n.listens
ORDER BY ABS(COALESCE(o.listens, 0) - COALESCE(n.listens, 0)) DESC
LIMIT 100

The log becomes the master dataset

The archive stops being the master dataset and becomes an audit copy; the log takes that role for 30 days. At 3,000 events per second and about 400 bytes each, 30 days with 3 replicas fills about 9.3TB of broker disk, which the team weighs against the 5-hour nightly job's compute and accepts. A replay runs at 120,000 events per second, and while it runs the output exists twice, so the serving store needs room for 2 copies of episode_listens, about 40GB each. A correction older than 30 days reads the archive through the same transform, across section 1's handoff seam.

• 104GB
• 18.5 h

The team wants to correct 60 days of history after a tagging bug, but the log retains 30 days. Which plan keeps the single-codebase property?

• Give the stream job's reader an archive mode that reads days 31 to 60 from files, hands off to the log at the archive's last offset, and run the same transform as v4.
• Write a batch job over the archive for days 31 to 60, let the stream job handle the last 30, and merge the 2 tables at read time for consumers.
• Raise retention to 60 days now, wait 30 days for the log to fill, then replay from the oldest offset with the fixed v4 job.
• Replay the 30 days available with v4 and scale the older counts by the correction ratio observed on the replayed window.

New checks replace the nightly overwrite

Before step 5 the team writes down what gets harder. With no nightly overwrite, nothing erases drift in the stream job's counts, so a once-free correctness check must be built: an hourly query comparing sampled episodes against raw event counts in the log. A bad deploy now reaches every consumer at once, so every change ships as a new vN replay beside the running version. Retention is now a correctness boundary too: an alert fires when the oldest event a planned correction needs reaches 25 days old, 5 days before the log deletes it, leaving time to replay or read the archive.

Two weeks after cutover, a creator reports an episode's total fell 3% overnight. Under the new design, which cause is possible, and which old-design cause is ruled out?

• Possible: the nightly recompute applied a rule the stream job lacks. Ruled out: a replay cutover, since replays only add listens.
• Possible: retention expired and deleted 30-day-old events, lowering stored counts. Ruled out: nothing from the old design.
• Possible: a new replay cut over with a stricter completed-listen rule. Ruled out: a nightly batch overwriting the speed layer's count.
• Possible: the speed layer reset at 02:00 before the batch view landed. Ruled out: a replay, since v3 writes the table being read.

About this lesson

This section is part of the Batch vs Streaming: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.