Spark Streaming vs Structured Streaming

The 2 word count listings look alike but count different things. reduceByKey on a DStream counts within each batch. A running total across batches needs updateStateByKey, which requires checkpointing, runs its update function for every key in every batch and drops a key when the function returns None. In Structured Streaming groupBy().count() is already a running count, and Spark keeps its state in the checkpoint location.

For custom per-key state, Structured Streaming has mapGroupsWithState and flatMapGroupsWithState since Spark 2.2. Since Spark 4.0 the guide encourages the newer transformWithState operator for complex stateful applications. Stream-stream joins exist since Spark 2.3, and for outer joins you must specify a watermark and event-time constraints.

Structured Streaming also has limits you will hit. On streaming Datasets, limit isn't supported, nor is taking the first N rows, and distinct isn't supported either. Sorting works only after an aggregation in Complete output mode. Actions such as count() and show() do not apply: you write to a sink, such as the console sink, instead.

A DStream window is defined by a window length and a slide interval, and both must be multiples of the batch interval. Windows are cut from when batches were processed. Event time is the time embedded in the data itself, and in Structured Streaming a window aggregation is a grouping on the event-time column, where each window is a group and a row can belong to several windows.

Late data is where that matters. Since Spark 2.1 Structured Streaming supports watermarks, which let you state how late data may be and let the engine clean up old state. The guarantee runs in one direction: data less than the threshold behind the latest event time is guaranteed to be aggregated, and data later than that may or may not be.

Both APIs can reach exactly-once, by different routes. With DStreams, the guarantee depends on the source and the output. Reliable receivers acknowledge a source only after the data is replicated, unreliable receivers can lose data on failure, and the Kafka Direct API, introduced in Spark 1.3, receives Kafka data exactly once. Output operations such as foreachRDD are at-least-once, so exactly-once end to end needs an idempotent update or a transaction identified by batch time and partition index.

Structured Streaming builds the guarantee into the engine. Sources are assumed to have offsets, the engine records the offset range of each trigger with checkpointing and write-ahead logs, and the sinks are designed to be idempotent for reprocessing. With replayable sources and idempotent sinks, the Structured Streaming guide states end-to-end exactly-once semantics under any failure. The exception you own is foreachBatch, which is at-least-once by default: use the batchId it passes you to deduplicate.

A DStream job is stable only while batch processing time stays below the batch interval. Take a 2 second interval and batches that need 3 seconds: each batch finishes 1 second later than the next one is due, so after 10 batches the job is 10 seconds behind and still falling back.

Structured Streaming handles overruns by rule. With no trigger set, the next micro-batch starts as soon as the previous one completes. With a fixed interval, if a batch overruns, the next starts immediately instead of waiting for the next boundary, and if no new data is available no batch runs. The default micro-batch engine reaches end-to-end latencies of about 100 ms at best. Continuous Processing, experimental since Spark 2.3, aims at about 1 ms with at-least-once guarantees for map-like queries. Real-time Mode, introduced in Spark 4.1.0, targets millisecond latency with the same APIs, for stateless queries in 4.1.0 and some stateful ones from 4.3.0.

A trigger that DStreams never had is availableNow: it processes everything available, possibly in several micro-batches, then stops. It suits jobs that start a cluster periodically, process what arrived since the last run and shut down. The older one-time trigger is deprecated in its favor.

A DStreams job in production usually carries more than a chain of transformations. Start with the state: every updateStateByKey or windowed reduce with an inverse function depends on checkpointing, and each one needs a Structured Streaming aggregation or stateful operator with a watermark that bounds how long state lives.

Then read the output. foreachRDD is at-least-once, so the job already carries an idempotent write or a transactional commit tagged with batch time and partition index. In Structured Streaming the equivalent is the built-in idempotent sinks, or foreachBatch with the batchId as the deduplication key. Last, find the clocks. A DStreams window counts time by when batches ran, so a rewrite to event-time windows changes which window a late record lands in, and the new output should be compared with the old before the old job is switched off.

Interviewers use the pair to see whether you know what a streaming engine guarantees. The strong answer names the model (RDD micro-batches vs an incrementally maintained query), says where event time and watermarks come in, and says what you must add yourself for exactly-once output. Be ready to say which trigger fits a job and what happens when a batch overruns.

Practice the Spark side with the PySpark interview questions and the PySpark practice problems. The design side, such as choosing batch or streaming for a pipeline, is covered by streaming system design interview questions and the data engineer interview prep hub.