Lambda Architecture
Concepts covered: Lambda Architecture
Three layers that answer one question
Lambda architecture computes every answer twice and merges the 2 results. The batch layer keeps an append-only master dataset of every event ever received and periodically recomputes a batch view from scratch over all of it. The speed layer processes events as they arrive and keeps a realtime view covering only the events the last batch run has not yet absorbed. The serving layer merges the 2 views at query time: a recomputed answer plus a thin slice of recent data.
- page_view_events (source, event feed)
- master_dataset (storage, append-only archive)
- batch_recompute (transform, nightly job)
- batch_view (storage, totals through cutoff)
- speed_layer (transform, stream job)
- realtime_view (storage, totals since cutoff)
- serving_merge (custom, query-time join)
- analytics_dashboard (consumer, reads merged totals)
- page_view_events -> master_dataset
- master_dataset -> batch_recompute
- batch_recompute -> batch_view
- page_view_events -> speed_layer
- speed_layer -> realtime_view
- batch_view -> serving_merge
- realtime_view -> serving_merge
- serving_merge -> analytics_dashboard
- Sources: page_view_events
- Batch layer: master_dataset, batch_recompute, batch_view
- Speed layer: speed_layer, realtime_view
- Serving: serving_merge, analytics_dashboard
A worked merge
Take a news site counting views per article. The batch job runs at 01:00 and recomputes every article's total from the master dataset up to midnight. The speed layer counts views per article since the last cutoff; once a batch view covering up to midnight lands, it drops its counts from before midnight and keeps the rest. At 15:00 the serving layer adds the 2, and article a-1101, published at 09:00, has no batch row yet.
| article_id | batch view (through 00:00) | realtime view (since 00:00) | served total |
|---|---|---|---|
| a-1042 | 1,208,311 | 4,517 | 1,212,828 |
| a-1077 | 88,402 | 12,960 | 101,362 |
| a-1101 | no row | 2,344 | 2,344 |
SELECT
COALESCE(b.article_id, s.article_id) AS article_id,
COALESCE(b.views, 0) + COALESCE(s.views, 0) AS served_views
FROM batch_view b
FULL OUTER JOIN realtime_view s
ON s.article_id = b.article_idLambda's correctness comes from the batch side of the merge. A speed-layer mistake, such as a view double-counted after a crash, survives only until the next batch run, because the batch view is recomputed from the master dataset without reading the speed layer's output. Whatever the realtime view gets wrong, the next recompute replaces, so the team can throw that view away at any time.
The constraints of 2012
Nathan Marz, whose stream engine Twitter open-sourced in 2011, laid out the design in a blog post that year and named it Lambda in his later book. It answered the limits of that era's engines: a crash could redeliver a message into a running count that then stayed wrong for good, stream state lived in process memory and vanished on restart, and no batch engine could recompute years of events fast enough to serve recent data, so recency needed a second path.
- Constraint in 2012
- A crash could replay a message into a count that had already absorbed it
- Stream state lived in worker memory and vanished on restart
- A full recompute took longer than the freshness consumers accepted
- The event feed kept hours of data, so history lived only in the archive
- Same constraint today
- Engines checkpoint state and read position together, so a restart resumes without double counting
- State is persisted to local disk and checkpointed, holding billions of keys
- Recompute is a replay of the log through the same job, at many times the live rate
- Logs retain weeks or months, so the log itself can be the history
Lambda bought correctness a single stream job could not deliver, and paid with 2 implementations of every rule. Once a stream job could survive a crash without corrupting its counts and the log could hold months of history, replaying the stream job itself could do the batch layer's recompute. That design is next.
A product manager wants views from known crawler IP ranges excluded from article totals starting today. Under the Lambda design above, what is the smallest change that makes served totals correct from tomorrow on?
- Add the filter to the speed layer only; tonight's batch recompute picks it up, since the master dataset holds every raw event.
- Add the filter to the batch recompute only, since the nightly batch view replaces whatever the speed layer got wrong.
- Add the filter to both the batch recompute and the speed layer, and accept mixed totals today until the 01:00 run rebuilds the batch view.
- Add the filter to the serving merge query, so both views are corrected at read time without touching either layer.
About this lesson
This section is part of the Batch vs Streaming: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.