Stateful vs Stateless Transforms

Concepts covered: Stream Processing

Transforms divide into two categories that matter much more in streaming than in batch. A stateless transform processes one event at a time and produces output that depends only on that event. A stateful transform produces output that depends on more than one event: a count, a sum, a window, a join with another stream. The category changes the cost, the complexity, and the failure-recovery story. In batch, both categories look about the same because the engine has all the data in memory at once. In streaming, the difference is structural and shapes nearly every design decision.

Stateless Transforms

Transform	What It Does	Why It Is Stateless
Filter (where clause)	Drops events that fail a predicate	Decision uses only the current event
Projection (select columns)	Reshapes the event without combining with others	Output is a function of the single input
Per-event enrichment from a static lookup	Joins with a small reference table loaded into memory	Reference data is constant; not derived from event history
Type conversion	Casts string to int, parses JSON	Single-event operation; no history needed
Field-level redaction	Hashes or removes PII fields	Per-event transformation

Stateless transforms are cheap and easy in both batch and streaming. They scale linearly with event count, recover trivially from failure, and require no state store. A streaming pipeline of pure stateless transforms behaves almost like batch in cost and complexity, only with a different rhythm. The interesting cases, the ones that justify the streaming infrastructure, are stateful.

Stateful Transforms

Transform	What It Does	Why It Is Stateful
GROUP BY aggregations	Counts, sums, averages over a key	Result depends on every event for that key seen so far
Windowed aggregation	Rolling counts over a time window	Result depends on which events fall inside the window
Stream-stream join	Matches events from two streams within a time window	Requires holding both streams in state until matched
Deduplication	Drops repeated events by key	Requires remembering keys already seen
Sessionization	Groups events into sessions by gap timeout	Each session is open until a gap closes it

How Streaming Engines Handle State

Modern streaming engines maintain state in a key-value store that is local to the worker for performance and snapshotted to durable storage for fault tolerance. Flink uses RocksDB local on the task manager and asynchronously checkpoints to S3 or HDFS. Spark Structured Streaming uses HDFS or S3 directly with a write-ahead log. Kafka Streams uses RocksDB plus a Kafka topic as the changelog. The pattern is the same: keep state hot for read performance, persist it for recovery. The cost is the dual storage layer, and the complexity is making the snapshot consistent with what has been emitted.

# A stateful streaming transform: rolling 5-minute count per page
from pyspark.sql.functions import window, count

stream = (
    spark.readStream
         .format('kafka')
         .option('subscribe', 'raw.clicks')
         .load()
         .select(parse_json('value').alias('event'))
)

windowed = (
    stream.withWatermark('event.timestamp', '10 minutes')
          .groupBy(
              window('event.timestamp', '5 minutes'),
              'event.page'
          )
          .agg(count('*').alias('clicks'))
)

# State per (page, window) is held until the watermark passes the window's end
query = (
    windowed.writeStream
            .outputMode('update')
            .format('delta')
            .option('checkpointLocation', 's3://chk/page_5min')
            .trigger(processingTime='30 seconds')
            .start('s3://lake/page_5min')
)

The withWatermark call is the bound on how long state is kept. Without it, state grows without limit because the engine cannot decide when a window is closed. With a 10-minute watermark, the engine drops state for windows whose end is more than 10 minutes behind the maximum event timestamp seen. Watermarks are the streaming equivalent of taking out the trash; without them, every state-laden pipeline eventually runs out of memory.

The Cost Difference Per Category

Stateless Streaming

Memory and disk usage scale with batch size, not history
Recovery: replay last batch or two; state is reconstructed from data
Schema changes: relatively safe; no embedded state to migrate
Cost premium over batch: roughly 2x to 5x

Stateful Streaming

Memory and disk usage scale with key cardinality and window size
Recovery: restore from checkpoint; in-flight state must be consistent
Schema changes: state migrations are required; not always trivial
Cost premium over batch: roughly 5x to 50x

Stateful Transforms in Batch

Batch handles stateful transforms naturally because the engine has all the data in memory at once. A GROUP BY in SQL is stateful in the engineering sense (the result depends on every row in the group), but the engine sees the whole input and produces the result in one pass. There is no checkpointing, no watermark, no out-of-order arrival to worry about. The cost is correctness without complexity. The downside is that batch cannot react to events in motion; it can only react to events that have already landed. Streaming pays for the ability to react in motion with the cost and complexity of state management.

Stateless transforms are cheap and recover trivially; they look the same in batch and streaming.
Stateful transforms drive most of the streaming cost premium; key cardinality and window size dominate.
Watermarks bound how long state is kept; without them, stateful pipelines run out of memory.

About this lesson

This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.