Unified Engines: Where Lines Blur

Concepts covered: Batch vs Streaming

One transform, bounded or unbounded input

A unified engine treats a batch as a stream that happens to end. The transform is written once as a graph of operators, filters, projections, grouped aggregations and joins, and the engine decides at launch whether to feed that graph a bounded input, such as a month of files, or an unbounded one, such as a live log. Only the launch parameters change between the 2 runs.

views = source("page_views")

per_article = (
    views
    .filter(lambda e: not e["is_crawler"])
    .group_by("article_id")
    .count()
)

run(per_article, mode="batch", input="archive/2026-10")
run(per_article, mode="stream", input="log/page_views")

A declarative query works the same way: the grouped count below fully describes the result in both modes, and the engine builds the same operator graph from it.

SELECT
    article_id,
    COUNT(*) AS views
FROM page_views
WHERE is_crawler = false
GROUP BY article_id

The engine unifies the language and operators: a developer learns 1 API, and an aggregation means the same thing over finished or ongoing input. It unifies testing, because the stream job can run over a bounded sample file and be checked against a hand-computed answer. It also unifies sources and sinks, since a connector for a bounded file and one for a log present the same interface to the operator graph.

Execution still differs

Execution is where the 2 modes part. In batch mode the query above reads every row, emits exactly 1 row per article, and exits. In stream mode it emits a row for a-1042 when the first view arrives, then a replacement row every time that count changes, for as long as the job lives. A consumer that appends each emitted row ends up with thousands of rows for 1 article, so it must replace by key. The query is identical, but batch mode writes a table once and stream mode writes a changelog that never ends.

aspect	batch run	streaming run
input	bounded, fully known before the job starts	unbounded, arriving while the job runs
operator lifetime	stage by stage, each finishing before the next	every operator alive at once, pipelined
state	sort and shuffle files for 1 job, discarded at exit	local stores, checkpointed, kept across restarts
state size bound	the input size	the distinct key count, growing until keys expire
failure recovery	rerun the failed stage from its materialized input	restore the last checkpoint, reread the log from that offset
output	the complete result, written once	a changelog of inserts and replacements, written continuously
resources	a burst, released at exit	a reservation held around the clock
ends when	the input is consumed	someone stops it

The state rows bite hardest. A batch grouped count over 30 days of views holds a count per article while it runs, then frees it. The streaming version holds that count for the job's whole life, often months, so an article published in January still occupies state in October whether anyone views it or not. A join between 2 streams must hold one or both sides in state until a match can arrive, where a batch join sees both sides complete in one pass. Stateful transforms are where the same code stops meaning the same cost.

For architecture, the unified engine shrinks Lambda's cost without settling the design question. With 1 codebase, a batch recompute beside a stream job is a second launch of the same program. Recompute versus replay becomes a storage question, whether history is replayed from the log or re-read from an archive, and section 4's migration puts numbers on it.

A team writes a running count of views per article, with no window, tests it over a 2-hour sample file, and deploys it in stream mode. Which runtime difference is most likely to surprise them in production?

The grouped count emits a different final number per article in stream mode than in batch mode for the same input, because streaming aggregations are approximate.
The stream run cannot be recovered after a crash, while the batch run can be rerun from its input files, so every crash loses the counts accumulated so far.
The stream job emits exactly 1 row per article once its input is fully read, so the sink can append the rows safely without replacing by key.
State grows with the number of distinct articles seen and does not shrink unless keys are expired, while the batch test held state only for the sample's articles and freed it at exit.

About this lesson

This section is part of the Batch vs Streaming: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.