Why Streaming Costs More
A batch job rents machines for the minutes it runs. A streaming job rents them for every minute of the day, including the hours when almost nothing arrives. That difference drives most of the cost gap.
| Line item | Nightly batch | Streaming | Ratio |
|---|---|---|---|
| Compute, 4M rows/day | 4 machines x 1 h = 4 machine-hours | 3 machines x 24 h = 72 machine-hours | 18x |
| Peak sizing | Sized for the whole day's rows in 1 h | Sized for the busiest second, idle the rest | n/a |
| Writes to curated layer | 1 commit per day | 1 commit per second = 86,400 | 86,400x |
| Always-on queue | None, reads yesterday's table | Queue cluster running 24 h | new line item |
| People | Failures found next morning | Pager at 03:00 for a stalled job | on-call rota |
The compute line surprises people in the other direction: streaming uses fewer machines at any moment than the batch job at its peak, because the work spreads across the day instead of piling into 1 hour. It still costs 18 times the machine-hours because those 3 machines never stop. Reserved pricing can shave the ratio, but the shape stays: streaming pays for idle time, and batch pays only while it runs.
3 places the premium hides
Idle capacity, commit overhead and on-call
- Idle compute: Machines sized for the busiest second run through the quietest hour. A pipeline whose input is 10x higher at noon than at 04:00 wastes most of its capacity most of the night.
- Per-write overhead: Each commit to the curated layer costs roughly the same whether it carries 1 row or 1M. Committing every second pays that cost 86,400 times a day, and readers then pay again to open thousands of tiny outputs.
- Operations: A nightly job that fails at 02:00 is rerun at 09:00 by whoever arrives first. A stream that stalls at 02:00 is 7 hours behind by 09:00, so someone carries a pager.
- 8,760 h
- 365 h
Pricing the delay
The premium is justified when a decision made from fresh data is worth more than the premium. Put a price on the delay. A fraud check that declines a stolen card at checkout saves the full order value; the same check run tomorrow morning saves nothing, because the goods shipped. A weekly revenue report that is 1 hour stale costs nothing, because nobody reads it until Monday.
| Consumer | Cost of 1 hour of staleness | Freshness that removes it | Pattern that earns its premium |
|---|---|---|---|
| Card fraud decision | Order value per fraudulent order | Seconds | Streaming |
| Warehouse stock on product page | Refund per oversold unit | Minutes | Micro-batch |
| Marketing spend dashboard | A mistimed budget shift, once a day at most | Hours | Hourly batch |
| Monthly finance close | Nothing | Days | Nightly batch |
Work the arithmetic for the stock example. The store oversells roughly 2 units per hour of staleness, and each oversell costs $15 in refunds and support time. Hourly batch leaves 1 hour of staleness: 48 oversells a day, $720 a day in cost. A 1 minute micro-batch leaves 1 minute of staleness: under 1 oversell a day, around $12. If moving from hourly batch to micro-batch adds $200 a day in compute, it pays back more than 3 times over. Per-event streaming at $2,000 a day would save the last fraction of an oversell and lose money.
- Latency is worth the premium when
- The consumer is a machine, or a person acting within minutes
- Each minute of staleness has a cost you can write down
- Input arrives steadily, so machines are rarely idle
- Latency is wasted spend when
- The consumer reads on a schedule: a daily report, a weekly review
- Staleness costs nothing until a threshold measured in hours
- Input arrives in daily dumps, so the stream waits on an upstream batch anyway
A team proposes replacing a nightly batch that feeds a weekly executive revenue deck with a streaming pipeline. The batch costs $40 a day; the stream estimate is $600 a day. The deck is opened on Monday mornings only. Which judgement is right?
- Accept: fresher data is always more accurate, and $600 a day is small for an executive audience.
- Reject: streaming cannot produce a weekly aggregate, so the deck would break.
- Accept: streaming removes the nightly failure risk, so the $560 premium is insurance.
- Reject: the consumer reads weekly, so the freshness gain has no value and the premium is $560 a day of pure cost.
About this lesson
This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.