Stream Processing

This one shows up as 'how would you flag a stolen card before the second swipe goes through?' or 'what changes when the data never stops arriving?' The interviewer wants 2 things: an accurate picture of what a stream is, and a justification for paying for it, because a stream costs more to build and to run than a nightly job, and choosing one without a reason is the most common way to lose this question.

What a stream is

A stream is an unbounded input. Events keep arriving, there is no last row, and the job processes each event, or a small group of them, shortly after it appears. Every stream runs from a producer through a broker to a consumer. A producer writes events: an app, a sensor, a database emitting its changes. A broker holds them, usually as an append-only log that keeps events for a retention period and lets several consumers read independently. The log is split into partitions so consumers can share the work, and order holds only within a partition, so all events for 1 key, such as every swipe on 1 card, go to the same partition. A consumer reads from the broker and, after doing its work, records how far it has read. Latency is measured in seconds or less, and that latency is the entire reason the design exists.

{
  "event_id": "evt_01HZX4K9",
  "event_type": "card_swipe",
  "card_id": "card_88213",
  "merchant_id": "m_4471",
  "amount_cents": 4999,
  "event_time": "2026-10-18T14:02:11Z"
}

Event time and processing time

Every event has 2 timestamps, and the interviewer is listening for whether you keep them apart. Event time is when the thing happened, written by the producer. Processing time is when the consumer saw it. A phone that lost signal in a tunnel can emit an event whose event_time is 4 minutes older than its arrival. Any count 'per 5 minute window' has to say which clock defines the window, and a stream that groups by event time has to decide how long it waits for stragglers before it publishes a number.

Event evt_103 is the late data case. If the window closed at 14:05 and emitted a count of 2, the true count is 3. The answer that scores has the window wait a grace period before emitting, and logs anything later and folds it into a correction rather than dropping it in silence. With a 2 minute grace, the window emits 2 at 14:07, and evt_103 at 14:07:40 turns that into a correction to 3. A 3 minute grace would have caught it, at the cost of every result landing a minute later. Give the grace period as a number in your answer, because every minute of it delays the result that streaming was supposed to make fast.

Retry in a stream

Consumers crash. When one restarts it resumes from the last position it saved in the broker, which means every event between that save and the crash is read a second time. The property that makes this harmless is a store that treats event_id as a key, so the second copy overwrites the first or is ignored, and a swipe never becomes 2 swipes. Name that in the answer and the retry follow-up is already closed.

Justifying the cost

What a stream costs

• Compute that runs around the clock, for an hour of 10 events as much as for an hour of 10 million
• A broker to operate, where someone has to watch retention and partitions as well as consumer positions
• Harder debugging: there's no input file to open and rerun, only a log position
• 2 clocks and late data inside every aggregate

What pays for it

• A decision whose value decays in seconds: a fraud block, an oversold item, a surge price
• An alert that is worthless if it lands an hour late
• A user-facing feature that reflects the user's own last action
• Events that are also written to storage, so batch jobs can reuse them later

• 7 days
• 16

The scored version

'A stream is unbounded input processed as it arrives, moving from a producer through a broker to a consumer. For fraud, the card terminal publishes a swipe event, a consumer scores it against the card's recent history and writes a decision in under 2 seconds, keyed by event_id so a replay after a crash cannot double count. Windows are defined on event time with a short grace period for late events. The cost is an always-on consumer and a broker to operate, and it is justified because an approved fraudulent swipe is a direct loss.' That answer defines the idea, applies it, handles both failure cases and justifies the spend, which is the full rubric.

About this lesson

This section is part of the How Data Moves: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.