When Would You Use Streaming?
Streaming is a cost paid to shorten the gap between an event and a decision made on it. Before paying, find out how short that gap has to be, and take the number from the consumer of the data rather than from the pipeline team. A fraud model approving a card swipe has about 300 ms before the terminal gives up. A finance analyst closing the month opens the revenue report once, on the 2nd business day. Both read the same transactions, and only 1 of them justifies a stream.
The 4 factors
Score every workload on these before choosing
- 1. Latency the consumer acts on: When does a person or a system do something different because the data arrived? Write it down: 300 ms for the fraud score, 2 seconds for a rider ETA, 6 hours for a demand forecast.
- 2. Cost of being late: A stockout alert delayed by an hour costs an hour of lost sales on that item. A weekly cohort chart delayed by an hour costs nothing, because nobody reads it until Monday.
- 3. Shape of the source: Order events and GPS pings arrive 1 at a time with timestamps and can be streamed, and so can change-data-capture rows. A nightly dump of a vendor database arrives whole at midnight, and no engine can stream data it cannot see until then.
- 4. Team and operations: A stream runs 24 hours a day, holds state in memory, and pages someone when it stalls. A batch job that fails at 02:00 reruns at 07:00 with no data lost. Ask who is on call and whether they have run a stateful job before.
The first factor sets the floor, and the way to use it is arithmetic. Write the SLA as the time from the event to the moment the consumer needs the result, then compare it with what batch can reach. An hourly job that runs for 10 minutes delivers a worst case of 70 minutes after the event. A daily job finishing at 06:00 for a report opened at 09:00 is early by 3 hours. Streaming is worth its cost when the SLA is shorter than the worst case of the fastest batch cadence you can afford.
| Workload | Latency acted on | Cost of being late | Source shape | Call |
|---|---|---|---|---|
| Card fraud score | 300 ms | fraud approved or a customer declined, per swipe | 1 event per swipe | stream |
| Rider ETA | 2 s | wrong ETA, cancelled trip | GPS ping every 4 s | stream |
| Warehouse stockout alert | 5 min | lost sales on the item until restocked | inventory change events | stream, or 30 s micro-batch |
| Ad click attribution | 1 hour | advertiser dashboard lags | click events | micro-batch, 5 min trigger |
| Monthly revenue close | 2 days | none within the day | ledger snapshot at midnight | daily batch |
The SLA column alone decides the first 2 rows and the last. The ad click row is the arithmetic above: an hourly job running 10 minutes lands 70 minutes after the event, past the 1 hour SLA, so it becomes a micro-batch, small batches run on a fixed trigger, here 5 minutes. The stockout alert is the row interviewers like, because 5 minutes is reachable 2 ways: a stream processing each event on arrival, or a micro-batch with a 30 second trigger. Both meet the number, so the team decides, and a warehouse team already operating the batch engine picks the micro-batch, which runs on it.
Put a number on the cost
The cost of streaming is compute that never stops. Take 50 million events a day. A daily batch job reads them in 40 minutes on 20 nodes. A streaming job on the same 20 nodes runs for all 24 hours, holding the cluster whether events arrive or not. The ratio between the 2 is about 36 to 1 before anyone adds the state store and checkpoint storage, let alone the engineer woken at 03:00. That ratio is the price, and factor 2 says whether being late costs more than it.
- 13 node-hours
- 480 node-hours
Practice: a request for real-time order counts
Hybrid answers score well when each half has a reason. Stream the order events for the wall screen with a 10 second trigger, and recompute the finance totals from the same log nightly as a batch. The SLA chooses the path for each consumer. One source feeding 2 cadences is the common shape, and the next section shows how to keep the 2 paths honest with each other.
About this lesson
This section is part of the Batch or Stream?: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.