Picking Batch or Streaming
Concepts covered: Batch vs Streaming
The choice comes down to 2 inputs: the consumer's freshness tier, which the previous section taught you to extract, and how much the owner is willing to pay and to operate. Batch wins by default. Streaming wins when the tier demands it and the budget accepts it. This section puts numbers on both inputs so the decision stops being a matter of taste.
Step 1: the tier narrows the field
| Consumer's tier | Choice | Why |
|---|---|---|
| Hours or more | Batch, nightly or hourly | One short run per window delivers the number before anyone reads it |
| Minutes | Batch on a short schedule | A job every 5 to 15 minutes meets the tier at a fraction of always-on cost |
| Seconds | Streaming | Only a consumer that is already running can react within seconds of each record |
| Sub-second | Streaming, with its data in memory | Even a write to a serving table may take too long at this tier |
Step 1 settles the decision for most consumers, because most consumers read reports, and reports sit in the hours tier. The contested cases cluster around the 1 minute boundary, where either shape can meet the tier and the second input decides.
Step 2: the cost you accept
Put a price on both shapes with one rate. Suppose the smallest machine that can run the signups counter costs $0.30 per hour. The nightly batch runs for 8 minutes, which is $0.04 per day. The stream consumer runs for 24 hours, which is $7.20 per day, roughly 180 times as much for the same 6 counted rows. The ratio shrinks as volume grows, because the batch run gets longer while the stream's hours stay fixed at 24, but at low volume streaming pays for 23 hours and 52 minutes of waiting.
- $2,628
- $14.60
Money is only part of the cost. A stream consumer is a process that can crash at 03:00 on a Sunday, and someone has to notice and restart it, because every minute it is down the live table drifts further from the truth. A batch job that fails at 02:05 can be restarted at 09:00 by whoever arrives first, and the output is still ready hours before the Monday meeting. Cost tolerance means both the bill and the willingness to run something that never stops.
- Do
- Write the consumer's freshness tier down before choosing a shape
- Default to batch, and make streaming earn its place with a seconds tier
- Price both shapes with the same hourly rate so the comparison is fair
- Feed a batch and a stream from one source when 2 consumers need 2 tiers
- Avoid
- Build streaming because the request contained the words real time
- Schedule a batch every 30 seconds to avoid admitting the case is streaming
- Judge cost by the bill alone and forget who restarts a crashed consumer at 03:00
- Let the loudest requester set the tier for every consumer of a table
Four cases, decided
| Use case | Who reads it, and when | Tier | Choice |
|---|---|---|---|
| Nightly sales report by region | Regional managers, at 08:00 | Hours | Batch at 02:00 over the previous day |
| Card swipe fraud score | The payment terminal, before it prints a receipt | Sub-second | Streaming, scoring each swipe as it arrives |
| Dashboard of failed logins | An on-call engineer, glancing every 10 minutes | Minutes | Batch every 10 minutes over the last 10 minutes |
| Monthly invoice generation | The billing system, on the 1st of each month | Hours or more | Batch on the 1st over the previous month |
Look at how the second column decides the third. Regional managers reading at 08:00 cannot act on anything fresher than a nightly run gives them. A payment terminal waiting for approval cannot act on anything slower than a stream delivers. The reader's rhythm sets the tier, and the tier sets the shape. The dashboard is the one case where budget matters, because a 10 minute batch and a stream both meet the tier, and the batch costs a fraction of the stream.
A support team wants to see a customer's total order count when that customer calls. Calls arrive all day. The team says an order placed 2 hours ago is fine to miss. Which choice fits?
- Streaming, because calls arrive all day and the agent needs the total within seconds of each call.
- Batch every 10 minutes, because order totals change often and a daily job would lag by a day.
- Batch every hour, because the team accepts a 2 hour age and an hourly run keeps the total under that limit.
- Streaming, because totals are per customer and a batch job cannot write one row per customer.
About this lesson
This section is part of the Batch vs Streaming: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.