Design for Both Batch and Stream

Lambda architecture answers a real tension: a stream gives a fast result that may be incomplete, and a batch gives a complete result that arrives late. The design runs both over the same events. The batch layer recomputes every result from the full history on a schedule, say hourly. The speed layer processes only the events since the last batch run completed. The serving layer answers a query by merging the 2 views, taking batch results up to the last complete hour and speed results after it. Nathan Marz described the pattern in 2011, and for years it was the standard answer when an interviewer asked for both freshness and correctness.

Every event takes both paths. The batch layer rewrites the batch view each hour from all history. The speed layer appends to the speed view as events arrive. The query reads batch up to the last complete hour and speed after it.

Order events (source, app)
Event log (queue, partitioned log)
Batch layer (transform, scheduled recompute); freshness: 1 hour; backfill: full history rerun
Speed layer (transform, incremental since last batch); freshness: seconds
Batch view (storage, complete hours)
Speed view (storage, live hours)
Serving query (consumer, merge of 2 views)

Order events -> Event log
Event log -> Batch layer
Event log -> Speed layer
Batch layer -> Batch view
Speed layer -> Speed view
Batch view -> Serving query
Speed view -> Serving query

Sources: Order events, Event log
Compute: Batch layer, Speed layer
Views: Batch view, Speed view
Serving: Serving query

The merge the serving layer runs

The 2 views overlap on purpose. The speed view holds every hour the stream has seen, including hours the batch has since recomputed, so the merge must cut the speed view at the batch boundary. Here the batch view holds complete hours of order counts per product, and the speed view holds the same shape built from the live log.

WITH batch_complete AS (
  SELECT
    max(hour_start) AS last_hour
  FROM batch_view
)
SELECT
  product_id,
  sum(order_count) AS order_count
FROM (
  SELECT
    product_id,
    hour_start,
    order_count
  FROM batch_view
  UNION ALL
  SELECT
    product_id,
    hour_start,
    order_count
  FROM speed_view
  WHERE hour_start > (SELECT last_hour FROM batch_complete)
) AS merged
GROUP BY product_id

With last_hour at 12:00, the query takes 395 from the batch view (120 plus 134 plus 141) and discards the speed rows for 11:00 and 12:00. Adding 57 from 13:00 gives a total of 452. Look at the speed view's 11:00 count of 130 against the batch's 134: the stream missed 4 late orders that the batch picked up an hour later. The merge hides that gap by preferring batch wherever batch exists. That is the correctness guarantee Lambda sells, and it works.

The dual-maintenance tradeoff

The cost sits inside the 2 boxes labeled batch layer and speed layer. Each holds its own copy of the business logic: which order statuses count, how an order maps to a product, how refunds subtract. The copies begin identical and drift. A status rule changes in the batch code on Tuesday, the stream version ships on Friday, and for 3 days the merged total disagrees with itself at the boundary hour. Every bug is fixed twice, every test is written twice, every schema change is deployed twice, and the 2 teams who own the halves argue about whose number is right.

2 codebases for 1 rule: the attribution logic lives in the batch job and again in the stream job
2 deploys per change: a rule change is live in 1 layer days before the other
2 test suites: the replay test for the stream and the golden-file test for the batch cover the same cases
1 boundary hour that disagrees whenever the layers are out of step

Convergence: one code path, 2 cadences

Jay Kreps published 'Questioning the Lambda Architecture' in 2014 with a simpler shape, later called Kappa: keep the log long enough, and when the logic changes, start a second instance of the same stream job from the beginning of the log, let it catch up, and switch readers to its output. Reprocessing becomes a replay in place of a separate batch program. The step after that came from engines that treat a batch as a bounded stream: the same job definition runs over yesterday's files as a batch and over the live log as a stream, with 1 planner, 1 set of tests and 1 deploy. The batch layer survives as a replay of the stream code.

Lambda (2 code paths)

Batch layer and speed layer written separately
Correctness comes from the batch overwrite
Reprocessing: rerun the batch job
Serving layer merges at the boundary hour
Drift between layers is a standing risk

Converged (1 code path)

1 job definition runs bounded or unbounded
Correctness comes from handling late data in the stream
Reprocessing: replay the log through the same job
1 output table, no merge
Needs log retention long enough to replay

The convergence has a condition. Replay works only when the log retains enough history for the rebuild, and when the stream handles late events well enough that the batch overwrite stops being needed. Both are testable, and the next section is about testing the second.

About this lesson

This section is part of the Batch or Stream?: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.