Streaming: Picture, Rhythm, Example
Concepts covered: Stream Processing
Streaming processing is the second basic rhythm. A streaming pipeline runs continuously. Each new event arrives at the source and flows through the transforms within milliseconds or seconds. There is no concept of a chunk and no concept of a scheduled wake-up. The pipeline is a long-running service, more like a web server than a script. The shape is more recent than batch in mainstream use, dating roughly from the rise of Apache Kafka in the early 2010s and the stream processors that grew up around it: Spark Streaming, Flink, Kafka Streams, Beam.
The shape of a streaming pipeline
| Element | What It Does | What It Looks Like |
|---|---|---|
| Source | Produces events continuously into a queue or log | Kafka topic, Kinesis stream, Pub/Sub topic |
| Consumer process | Reads events as they arrive, applies transforms, emits results | Long-running JVM, Python service, Spark cluster |
| State store | Holds running totals, windows, joins between events | RocksDB on local disk, in-memory cache, external KV store |
| Output sink | Where transformed events land for downstream consumption | Another Kafka topic, a database, a feature store |
A live event feed
The canonical example is a live event feed. A user clicks a button on a website. The click is recorded as an event, sent to a Kafka topic, and within a few seconds shows up on an internal dashboard, a fraud system, an ad attribution model, and a customer support tool. The same event triggers downstream consequences without waiting for a 2am run or for anyone to come in the next morning.
# Sketch of a streaming consumer in Kafka Streams style
from confluent_kafka import Consumer
consumer = Consumer({
'bootstrap.servers': 'kafka.internal:9092',
'group.id': 'click-aggregator',
'enable.auto.commit': False
})
consumer.subscribe(['raw.clicks'])
running_totals = {}
while True:
msg = consumer.poll(timeout=1.0)
if msg is None:
continue
event = parse(msg.value())
running_totals[event.page] = running_totals.get(event.page, 0) + 1
if should_flush(event):
write_to_dashboard(running_totals)
consumer.commit(msg)The shape is an infinite loop. A consumer reads the next event, updates state, possibly emits an output, commits the offset, and goes back to read the next event. The loop has no return value and no scheduled end; the process is started once and expected to keep running. Stopping it is a deliberate operational action, not the natural conclusion of the work.
Why streaming costs more
Streaming pays for compute around the clock. The Kafka consumer process is up at 3am whether or not events are arriving. The Flink cluster is provisioned for peak load even at off-peak hours. State stores like RocksDB hold data in memory or on local disk that costs money to keep around. There's no equivalent of the batch idle period. Cost is roughly proportional to peak event volume times time, instead of total event volume divided by amortized run time. For low-volume topics this can still be cheap, but it never approaches the cost-per-event of a well-amortized batch job.
- Batch cost profile
- Compute is on for the run, off the rest of the day
- Cost scales with the size of each chunk plus overhead per run
- Idle hours are free; ramp-up amortizes across the chunk
- A 1 percent traffic dip lowers cost the next night
- Streaming cost profile
- Compute is on every minute of every day
- Cost scales with provisioned capacity, not actual traffic
- Idle hours still cost the same as peak hours
- A 1 percent traffic dip lowers nothing; capacity is fixed
Common streaming use cases
| Use Case | Why Streaming Fits | Tolerable Latency |
|---|---|---|
| Fraud detection | Decisions must happen before the transaction settles | Sub-second to a few seconds |
| Live operational dashboards | Operators react to events as they happen | Seconds to a minute |
| Real-time personalization | User session is short; recommendations must adapt within the session | A few hundred milliseconds |
| IoT telemetry | Volume is too high to batch economically; alarms are time-critical | Seconds to a minute |
| Change data capture (CDC) | Change data capture (CDC) turns each row write in an operational database into an event stream; downstream replicas must reflect upstream changes within seconds | Seconds to a few minutes |
Streaming is not instant
The latency win over batch is real but bounded. A streaming pipeline still has to read, transform, and write each event, and physics imposes a floor on how fast that can happen. Network hops, serialization, state lookups, and downstream writes each add milliseconds. A typical end-to-end streaming latency in production is between 100 milliseconds and several seconds. Anything sub-100ms requires careful engineering and dedicated hardware, and the marketing word real-time says nothing about this floor.
About this lesson
This section is part of the Batch vs Streaming: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.