Stateful vs Stateless Transforms

A stateless transform produces its output for a record from that record alone: each row goes in, its row comes out, and the transform remembers nothing. A stateful transform's output depends on records it has already seen, so it needs memory of the past to produce the present. One question sorts any transform: does its output need earlier records?

Transform	Needs earlier records?	Class
Parse event JSON into columns	No	Stateless
Filter out internal test users	No	Stateless
Look up country from a static IP table	No	Stateless
Count orders per city per 5 minute window	Yes, every order in the window	Stateful
Drop an event whose id was seen in the last hour	Yes, every id for 1 hour	Stateful
Match each refund to its original payment	Yes, every unmatched payment	Stateful
Running lifetime spend per customer	Yes, all history	Stateful

The right table cannot be produced one row at a time. When e1 arrives, the pipeline knows Austin has 1 order so far; it must hold that partial count somewhere until 10:05, when the window closes and the row is final. In a batch job this is a GROUP BY, and the state lives in the memory of a run that ends in minutes. In a stream the job never ends, so state lives in a store that must survive for as long as the window is open, for every open key at once.

SELECT
  city,
  count(*) AS orders,
  sum(amount) AS total
FROM raw.order_events
WHERE event_ts >= TIMESTAMP '2026-10-27 10:00:00'
  AND event_ts < TIMESTAMP '2026-10-27 10:05:00'
GROUP BY city

State costs memory and time

State costs memory in proportion to the number of open keys times the size of each key's partial result, and it costs time to protect that memory. If the stream tracks 5 minute windows for 10,000 cities, the state is small. If it drops duplicate event ids seen in the last 24 hours at 2,000 events a second, the state holds 172.8M ids, several gigabytes, and every incoming event checks against it. The stateless transform next to it holds nothing and scales by adding machines; the stateful one scales by splitting keys across machines, and each key must land on the same machine every time.

Recovering state after a crash

A stateless job that dies restarts, reads the last position it recorded on the queue, and continues; it lost nothing because it held nothing. A stateful job that dies loses every partial count in memory. Without protection, the 10:00 to 10:05 Austin window restarts at 0 part way through, and the final row is wrong with no error to show for it. The protection is a checkpoint: every few seconds the job writes its state and its queue position together to durable storage, so a restart loads the snapshot and replays only the events after the saved position.

Checkpointing is itself a cost: the state is written over and over, and the write must be frequent enough that replay after a crash is short, but not so frequent that writing state crowds out processing events. A job with 8GB of state checkpointing every 10 seconds moves 69TB a day to storage for protection alone.

Classify this rule and say what it needs: flag any payment card used in 3 different cities within 10 minutes.

Stateless: each payment carries its own city, so the transform compares the city on the row to the previous row.
Stateful: the job keeps the cities seen per card for the last 10 minutes and checkpoints that store so a restart does not forget a card's recent cities.
Stateful: the job keeps every payment ever seen so any card's full history is available.
Stateless: the check can be done by a lookup against a static table of known cities.

About this lesson

This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.