Picking Batch or Streaming
Concepts covered: Batch vs Streaming
The 3 cases below are small and deliberately simple so the choice between batch and streaming is easy to see. Real production decisions are messier, but the same questions apply: what does the consumer need, when do they need it, and what does each option cost.
Case 1: a marketing team's daily signup count
The marketing team wants a chart of new signups by country, by day, for the trailing 30 days. The chart is read once a morning at the marketing standup. The numbers don't change after the day closes. The consumer is patient: yesterday's number is fine, today's morning number is a bonus. The freshness tier is 4 (daily). The right architecture is a nightly batch that aggregates the day's signups and writes a small partition. Streaming would work but cost much more for no extra value the consumer cares about.
Case 2: a fraud team's suspicious transaction alert
The fraud team wants to be alerted within seconds when a card has more than 5 distinct merchants in the last minute. The decision is whether to freeze the card before another transaction lands. A nightly batch produces the answer the next morning, when the card has already been used 20 more times. An hourly batch produces the answer up to an hour late, which is far too slow. The freshness tier is 1 or 2. The right architecture is a streaming consumer that maintains a per-card sliding window over recent transactions. The cost is real, but the alternative is unacceptable.
Case 3: an ad spend dashboard that updates hourly
The growth team wants a dashboard of cost per acquisition by ad campaign, updated hourly during business hours. The team uses the dashboard to pause underperforming campaigns within the same workday. Daily is too slow; sub-second is overkill. The freshness tier is 3 (same day). The right architecture is an hourly batch that runs at the top of every hour, reads the last hour of clicks and conversions, and writes the result. Streaming would work but cost 2 or 3 times as much. Daily would leave the team paying for bad campaigns until tomorrow.
| Case | Tier | Right Choice |
|---|---|---|
| Daily signups dashboard | Tier 4 (daily) | Nightly batch; cheapest, simplest, fits the consumer |
| Fraud transaction alert | Tier 1 or 2 (seconds) | Streaming; the latency justifies the cost |
| Ad spend dashboard | Tier 3 (same day) | Hourly batch; balances freshness with cost |
The 3-question test
If the answers point at tier 4 or 5, batch wins on cost and simplicity. If the answers point at tier 1 or 2, streaming wins because batch cannot meet the freshness floor. Tier 3 is the hard case: an hourly batch usually beats a streaming pipeline on cost while still meeting the consumer's need, but the right answer depends on the volume and the existing infrastructure. Asking the 3 questions in order resolves the choice in nearly every case.
- Batch wins when
- Consumer reads on a schedule (morning standup, end-of-day report)
- Freshness tolerance is hours or days
- Cost per dollar of value matters more than latency
- Failure recovery means rerunning a clean partition
- Streaming wins when
- Consumer reacts to events as they happen
- Freshness tolerance is seconds or single-digit minutes
- A late answer is worse than no answer
- Volume is high enough that buffering hours of data hurts
Most platforms need both
Most companies need batch for almost everything and streaming for a few specific consumers. Treating the 2 as competing philosophies is a mistake; they're tools with different cost profiles, suited to different freshness tiers, and a mature data platform uses each where it fits. Deciding which one to use is a freshness conversation followed by a cost conversation, with the tier named before the tool.
- Batch covers freshness tiers 3 to 5 at the lowest cost.
- Streaming covers tiers 1 and 2, and it is the only option when batch cannot meet the freshness floor.
- Running the 3-question test answers nearly every batch-vs-streaming decision in practice.
- Do
- Default to batch and graduate to streaming for the specific consumers that need it
- Translate any real-time request into a numeric freshness tier before picking an architecture
- Name the cost difference explicitly so consumers can opt in or out of the tier they think they want
- Avoid
- Build streaming for everything because it sounds modern; the cost compounds
- Build batch for tier 1 needs because it is simpler; consumers will work around the pipeline
- Skip the freshness conversation; tools chosen without it are usually the wrong tools
About this lesson
This section is part of the Batch vs Streaming: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.