Batch vs Streaming
This is the decision question, and it comes in 1 disguise after another: 'should the signups dashboard be real time?', 'would you stream the warehouse scans?', 'the product manager wants live numbers, what do you build?' The interviewer is scoring 1 move above all others: whether you ask about freshness before you choose. Answering 'streaming' in the first 3 seconds, because it sounds current, loses the question even when streaming turns out to be the right call.
The question to ask back
'How stale can this data be before someone makes a worse decision because of it?' That question turns a preference into a number. A finance team closing the month tolerates a day. A product manager watching a launch tolerates 15 minutes and would act no differently with 15 seconds. A fraud model tolerates 2 seconds. Once the number exists the choice follows from it, and the second question, 'who or what reads the output?', settles whether the consumer is a human refreshing a page or a system taking an action on its own.
- Batch fits when
- Freshness of an hour or more is acceptable
- The consumer is a report, a model training run or an invoice
- The computation needs the whole day to be correct, like a daily total
- Input arrives on a schedule anyway, such as a nightly file
- Correctness and rerunability matter more than speed
- Streaming fits when
- Freshness under a minute changes a decision
- The consumer is a system acting automatically
- The value of each event decays in seconds
- The source already emits events
- The team can operate a broker and an always-on consumer
| Request | Freshness needed | Consumer | Choice | Why |
|---|---|---|---|---|
| Monthly revenue close | 1 day | Finance analysts | Nightly batch | Totals need the full day, and any date must be rerunnable |
| Launch day signups dashboard | 15 min | Product team watching a page | Batch every 15 min | Humans act no faster than that, and no broker is needed |
| Card fraud decision | 2 s | Approval system | Stream | The event's value is gone once the swipe is approved |
| In-stock badge on a product page | 30 s | Web page, millions of visitors | Stream | Overselling costs refunds and trust |
| Nightly model training set | 24 h | Training job | Nightly batch | The job reads a bounded set once |
The middle option
Between a nightly job and a stream sits a batch that runs every 5 or 15 minutes. It uses the nightly job's query with a smaller window and replaces only its own small slice on rerun. It needs no broker, and once the number is on the table it covers a large share of 'real time' requests. Offering it shows the interviewer you reach for the cheapest design that meets the stated need.
The answer pattern
5 moves, in order
- Restate the need with a number: 'you want signups within 15 minutes of happening.'
- Name the consumer: 'a product team refreshing a page during launch week.'
- Choose, and tie it to the number: 'a batch every 15 minutes reads the same signup table the nightly job reads.'
- Name the cost of the alternative: 'a stream would need a broker and an always-on consumer for a page nobody refreshes faster.'
- Name where your choice breaks: 'if the ask drops under a minute, I would move signups to a stream.'
A worked exchange
Interviewer: 'Marketing wants to see campaign clicks as they happen.' You: 'How soon after a click does someone need to see it, and what will they do with it?' Interviewer: 'Within a few minutes. They pause campaigns that are failing to convert.' You: 'A human pausing a campaign acts on 5 to 15 minutes of data, so every 5 minutes I would run a batch that counts clicks per campaign in 5 minute buckets, replacing each bucket on rerun. No broker, and the same query as the daily report with a smaller window. If marketing later automates the pause, the consumer becomes a system and the need drops under a minute; at that point the clicks become a stream.' The interviewer got the number and the consumer first, then the choice with the cost it avoids, and finally the point where the design would switch. That covers every box on the sheet.
About this lesson
This section is part of the How Data Moves: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.