IntermediatePipeline Architecture · 35 min · 2 challenges

Batch or Stream?: Intermediate

Batch or stream is the first fork in every pipeline design interview, and interviewers use it to separate candidates who chase buzzwords from candidates who reason from requirements. The trap is accepting the question as asked. Strong candidates interrogate the latency requirement before choosing anything: what decision does this data drive, and how stale can it be before that decision changes? This lesson works through the five follow-ups that decide the round: when streaming is actually justified, how to design for batch and stream together, what happens to late-arriving events, micro-batch versus true streaming, and how each architecture recovers when a job dies mid-run.

What you will be able to do

Anchor the batch-or-stream decision on latency SLAs and the cost of staleness, not on what sounds modern

Handle late-arriving data with watermarks and reconciliation instead of assuming event time equals processing time

Choose between micro-batch and true streaming and explain each architecture's failure recovery story

When Would You Use Streaming?

Daily Life

Interviews

The Decision Framework

The interviewer wants to hear a number, not a vibe. Your first sentence should be: "What's the latency SLA?" Not "do we want real-time" - that's a wish, not a requirement. If they say "daily is fine," you just eliminated streaming from the conversation. That's a senior signal - you saved six months of unnecessary infrastructure.

There are exactly four factors that drive the batch-vs-stream decision. Your answer should walk through all four. Miss any one of them and the interviewer knows you haven't built this in production.

Factor	Favors Batch	Favors Streaming
Latency SLA	> 1 hour acceptable	< 5 minutes required
Data Volume	Bounded datasets, periodic loads	Unbounded, continuous arrival
Processing Complexity	Complex multi-pass joins, ML training	Filters, aggregations, enrichment
Cost Tolerance	Tight budget, shared cluster	Budget for always-on infra

TIP

The interviewer is scoring you on whether you ask clarifying questions before picking an architecture. "What's the latency requirement?" and "What's the event volume?" immediately signal senior thinking. Candidates who jump straight to "I'd use Kafka and Flink" cap themselves at mid-level.

Here's the trap: volume alone doesn't decide this. A 10TB nightly load is a perfect batch job. A 100-event-per-second click stream that needs sub-second fraud detection is streaming. The interviewer wants to hear you reason about the combination of latency AND volume AND complexity - not just pick the biggest number.

Real Interview Questions

▸A common prompt: "Design a fraud detection pipeline" - the latency SLA is milliseconds, this is streaming. Say that immediately.
▸Another common prompt: "Build a driver earnings report" - daily aggregation, this is batch. Don't overcomplicate it.
▸The trap: both involve high volume, but latency requirements are totally different. The interviewer wants to see you anchor on latency, not volume.

The meta-skill the interviewer is testing: can you push back? When a stakeholder says "we need real-time dashboards," your answer should be: "What decision changes if the data is 5 minutes old? What about 1 hour old?" Most analytics dashboards work perfectly fine with 15-minute refresh cycles - and that's a micro-batch job, not a streaming pipeline. Saying this out loud is a senior signal.

•Streaming Justified

Fraud detection (ms latency)
Real-time pricing/surge
Live operational alerts
Session-based personalization

•Batch Sufficient

Daily financial reports
ML model training
Historical analytics
Data warehouse loads

KEY TAKEAWAYS

Start with the latency SLA - a number, not a feeling. This is the first thing out of your mouth.

Walk through all four factors: latency, volume, complexity, cost

Most "real-time" requests are actually fine with near-real-time batch - say this and explain why

Asking clarifying questions before choosing architecture is the #1 senior signal interviewers report

Design for Both Batch and Stream

Daily Life

Interviews

Lambda Architecture in Practice

Here's the reality the interviewer is testing: production data platforms almost always need both batch and streaming. Batch gives you correctness. Streaming gives you speed. The question is how you merge them without creating a maintenance nightmare. Your answer should acknowledge both sides and then explain the merge strategy.

The Lambda architecture formalized this pattern: a speed layer (streaming) serves approximate results immediately, while a batch layer reprocesses everything for accuracy. A serving layer merges both views. The interviewer wants to hear you name "Lambda" explicitly and then immediately flag its biggest weakness: dual maintenance.

Speed (approximate, fast)

Batch (accurate, delayed)

Serving

Flink

speed layer

Spark

batch layer

Druid

serving layer

Consumer

dashboard

Lambda architecture: the speed layer serves approximate results now; the batch layer reprocesses everything for correctness; the serving layer merges both. The cost is two codepaths for the same business logic.

	# Lambda architecture - simplified merge logic
	# Speed layer: Kafka -> Flink -> speed_table (approximate)
	# Batch layer: S3 -> Spark -> batch_table (accurate)

	-- Serving layer query
	SELECT
	COALESCE(batch.user_id, speed.user_id) AS user_id,
	CASE
	WHEN batch.updated_at >= speed.event_time
	THEN batch.total_orders -- batch caught up
	ELSE batch.total_orders + speed.incremental_orders
	END AS total_orders
	FROM batch_table batch
	FULL OUTER JOIN speed_table speed
	ON batch.user_id = speed.user_id;

The follow-up will be: "What happens when batch catches up?" Your answer: the speed layer results must be replaceable. The streaming layer should produce deltas or increments - not recompute totals - so the batch layer can cleanly supersede it. If you say "we just overwrite," the interviewer will probe for race conditions. Have the COALESCE pattern ready.

Eventual Convergence: The Concept

▸The speed layer and batch layer WILL disagree temporarily - that's by design. Say this.
▸Convergence means: once batch reprocesses, both layers produce identical results
▸Red flag: if your streaming logic diverges from your batch logic, convergence breaks permanently
▸Senior signal: mention that many teams use the same processing code for both layers to prevent drift

TIP

Large ride-sharing platforms use a Lambda-style pattern where streaming produces incremental metrics and a nightly batch job reconciles everything. Drop this reference in your answer - it shows you've studied real production systems, not just read the Wikipedia page on Lambda architecture.

The trap the interviewer is setting: Lambda sounds elegant on a whiteboard. The real cost is maintaining two codepaths. Every business logic change ships to both the batch and streaming layers. Teams that ignore this end up with a speed layer that produces different numbers than the batch layer, and nobody trusts either. Acknowledge this cost unprompted.

•Lambda Benefits

Low-latency results available immediately
Batch layer guarantees correctness
Can reprocess full history
Tolerates streaming failures gracefully

•Lambda Costs

Two codepaths to maintain
Merge logic adds complexity
Higher infrastructure cost
Risk of divergent results

KEY TAKEAWAYS

Lambda = speed layer for freshness + batch layer for accuracy + serving layer to merge

The interviewer's follow-up will be about convergence: batch results must supersede streaming results

The real cost is maintaining two codepaths with identical business logic - flag this before they ask

If you can't afford dual maintenance, mention Kappa as the alternative (advanced topic)

What About Late Data?

Daily Life

Interviews

Watermarks, Windows, Late Arrivals

This is the #1 follow-up question after you propose a streaming architecture. Interviewers commonly ask some version of: "A user clicks at 11:59 PM but the event arrives at 12:03 AM. Which day does it belong to?" If you've already closed the daily window at midnight, you've got a problem. The interviewer wants to hear three words: watermark, allowed lateness, dead-letter.

Start your answer here: a watermark is the system's estimate of "how far behind is the latest event I haven't seen yet?" It's a threshold. Any event with a timestamp before the watermark is considered late. Your answer should define this crisply in one sentence before diving into configuration details.

	# Flink watermark strategy - allow 5 minutes of lateness
	from pyflink.common.watermark_strategy import WatermarkStrategy
	from pyflink.common.time import Duration

	watermark_strategy = (
	WatermarkStrategy
	.for_bounded_out_of_orderness(Duration.of_minutes(5))
	.with_timestamp_assigner(MyTimestampAssigner())
	)

	# Events up to 5 min late: included in the window
	# Events > 5 min late: dropped or sent to side output

The Scenario Interviewers Test

▸You compute hourly_revenue for 2-3 PM = $10,000
▸A $500 transaction from 2:45 PM arrives at 3:20 PM
▸If the window is already closed, that $500 is lost from the aggregate
▸Downstream dashboards now show $10,000 - permanently wrong
▸With a 30-minute allowed lateness, the window stays open and the $500 is included. State this tradeoff explicitly.

The follow-up will be: "How do you set the allowed lateness?" Your answer should frame this as a business tradeoff, not a technical default. Say: "Allowed lateness is a business decision - what's the cost of missing a late event vs. the cost of holding windows open longer and consuming more memory?" That framing is what separates strong from average candidates.

Window Type	Use Case	Late Data Behavior
Tumbling (fixed)	Hourly aggregates	Late events miss the window boundary
Sliding (overlapping)	Moving averages	Late events can update multiple windows
Session (gap-based)	User sessions	Late events can extend or merge sessions

Session windows deserve special attention because they're the most interview-relevant. Interviewers frequently ask about session-based aggregation. A session window has no fixed boundary - it closes after a gap of inactivity. The trap: late events within the gap extend the session, which can cascade and merge two previously separate sessions. If you mention this merge behavior unprompted, the interviewer knows you've built this in production.

TIP

When the interviewer asks about late data, your answer framework is: (1) watermark strategy and how you chose the bound, (2) allowed lateness duration and the business tradeoff, (3) what happens to data that arrives after the lateness window - is it dropped, logged, or sent to a dead-letter queue? Hitting all three is the complete answer.

KEY TAKEAWAYS

Watermarks estimate progress: events before the watermark are considered late. Define this in one sentence.

Allowed lateness is a business tradeoff - frame it that way, not as a technical config

Session windows are interview favorites - mention that late events can merge sessions before they ask

Always define a dead-letter strategy for events that arrive after allowed lateness

Micro-batch or True Streaming?

Daily Life

Interviews

Spark Structured Streaming vs Flink

The interviewer is testing whether you pick a framework based on requirements or based on resume keywords. "Streaming" is a spectrum, not a binary. On one end, Spark Structured Streaming processes data in small batches (micro-batches) with a latency floor around 100 milliseconds. On the other end, Apache Flink processes each event individually. Your answer should match the framework to the SLA, not to your personal preference.

•Micro-batch (Spark SS)

Latency floor ~100ms (typically 1-10s)
Familiar Spark API and ecosystem
Easier for teams already using Spark batch
Simpler state management

•True Streaming (Flink)

True per-event, sub-millisecond latency
Advanced windowing and event-time processing
Better for complex stateful computations
More powerful exactly-once guarantees

	# Spark Structured Streaming - micro-batch trigger
	from pyspark.sql import SparkSession

	spark = SparkSession.builder.getOrCreate()

	df = (spark.readStream
	.format("kafka")
	.option("kafka.bootstrap.servers", "broker:9092")
	.option("subscribe", "events")
	.load())

	# Trigger every 10 seconds - this is your latency floor
	query = (df.writeStream
	.trigger(processingTime="10 seconds")
	.outputMode("append")
	.format("delta")
	.start("/output/events"))

The trigger interval in Spark SS is the config the interviewer will probe. A 10-second trigger means your minimum end-to-end latency is 10 seconds plus processing time. You can lower it, but below 100ms you're fighting scheduler overhead and the batch paradigm breaks down. Say this - it shows you understand the framework's limits, not just its API.

State management is where the frameworks diverge most, and where the interviewer will dig if you claim Flink experience. Spark SS stores state in a state store (backed by HDFS or RocksDB). Flink manages state natively with RocksDB and snapshots it during checkpoints. For simple aggregations, both work fine. For complex event processing - pattern matching, session windows with merging - Flink's state model is significantly more expressive. Know which one your use case actually needs.

Consideration	Spark Structured Streaming	Apache Flink
Latency	100ms - 10s typical	< 10ms per event
State Backend	HDFS / RocksDB state store	Native RocksDB, incremental checkpoints
Learning Curve	Low if team knows Spark	Steeper, different programming model
Windowing	Basic tumbling/sliding	Advanced: session, custom triggers
Ecosystem	Delta Lake, MLlib, managed Spark platforms	Kafka tightly integrated, Flink SQL

TIP

Red flag: picking a framework and defending it blindly. Senior signal: "For this use case with a 30-second latency SLA and our existing Spark infrastructure, Spark Structured Streaming is the pragmatic choice. If the SLA drops below 1 second, we'd evaluate Flink." That conditional reasoning is what interviewers score highest.

The 80/20 Rule for Interviews

▸80% of 'streaming' use cases in production run on micro-batch with 1-60 second triggers
▸True per-event streaming is reserved for fraud detection, real-time bidding, and operational alerting
▸If your use case tolerates 10-second latency, micro-batch is dramatically simpler to operate. Say this.

KEY TAKEAWAYS

Micro-batch (Spark SS) has a ~100ms latency floor - fine for most use cases. State this tradeoff.

True streaming (Flink) wins when you need sub-10ms latency or complex stateful processing

Match the framework to the SLA and the team's skills - not to your personal resume

80% of production streaming is actually micro-batch. Saying this is a senior signal.

How Do You Handle Failures?

Daily Life

Interviews

Exactly-Once, Offsets, Checkpoints

Every streaming system will fail. The interviewer knows this. The question isn't whether failures happen - it's whether your pipeline produces correct results when they do. Your answer framework: start with delivery guarantees, then explain offset management, then describe your idempotency strategy. Hit all three and you've covered the full rubric.

Guarantee	Behavior	Use Case
At-most-once	Process event, then commit offset. Lost events on failure.	Logs, metrics where some loss is acceptable
At-least-once	Commit offset after processing. Duplicates on failure.	Most pipelines - deduplicate downstream
Exactly-once	Atomic commit of offset + output. No loss, no duplicates.	Financial transactions, billing

The interviewer wants to hear that at-least-once is the production default. Not exactly-once. The trap is answering "exactly-once" because it sounds correct. The real answer: at-least-once with idempotent sinks. If you process the same event twice, the result doesn't change. Upserts instead of inserts. Deduplication keys. This is what production systems actually use.

	# Kafka offset management - at-least-once pattern
	from kafka import KafkaConsumer

	consumer = KafkaConsumer(
	'events',
	bootstrap_servers='broker:9092',
	enable_auto_commit=False, # Manual commit!
	group_id='my-consumer-group'
	)

	for message in consumer:
	try:
	process(message) # Step 1: process
	write_to_sink(message) # Step 2: write output
	consumer.commit() # Step 3: commit offset
	except Exception:
	# On failure: offset NOT committed
	# Next poll re-delivers this message
	# -> at-least-once (possible duplicate)
	log_error(message)

What Interviewers Ask: Recovery

▸Flink snapshots all operator state to durable storage at checkpoint intervals
▸On failure, the job restarts from the last successful checkpoint
▸Default checkpoint interval: 10 seconds - the interviewer may ask you to tune this
▸Checkpoint duration should be << checkpoint interval, or you create backpressure

Offset management is the follow-up trap. Auto-commit (Kafka's default) commits offsets on a timer, regardless of whether processing succeeded. This means: crash after auto-commit but before writing to the sink = lost event. The interviewer wants to hear "disable auto-commit, use manual commits after successful processing." If you mention enable_auto_commit=False unprompted, that's a strong signal.

TIP

"How do you handle exactly-once?" is a trap question. The correct answer: "You usually don't. You design for at-least-once with idempotent sinks. True exactly-once requires transactional coordination between source and sink, which most systems can't provide end-to-end." Saying this confidently is a senior signal.

KEY TAKEAWAYS

At-least-once + idempotent sinks is the production default. Lead with this, not exactly-once.

Disable auto-commit; commit offsets only after successful processing and writing

Checkpoints snapshot operator state for recovery - know how to tune the interval

True exactly-once requires transactional coordination end-to-end, which is rare in practice

❯❯❯PUTTING IT ALL TOGETHER

> The interviewer says: "Design a pipeline for our payments product. We need fraud scoring on each transaction and a daily earnings report for merchants. The PM says everything should be real-time." You have a whiteboard and about fifteen minutes, and every follow-up will probe late data, framework choice, and what happens when the job crashes.

Split the request by latency SLA before drawing anything: fraud scoring is a millisecond decision so it is streaming, and the merchant earnings report is a daily aggregation so it is batch, which shows you anchor on latency rather than on the fact that both are high volume.

Answer the PM's "everything real-time" with the pushback question, "what decision changes if this is 5 minutes old, or an hour old?", and land most of the reporting surface on a 15-minute micro-batch refresh instead of a streaming pipeline nobody needs.

Where fraud scores and the daily report must agree, name Lambda architecture, sketch the COALESCE serving-layer merge, and say unprompted that the speed layer emits increments so the batch layer supersedes it cleanly, and that dual maintenance of two codepaths is the real cost.

Pre-empt the late-data follow-up with the 11:59 PM click arriving at 12:03 AM: define the watermark in one sentence, set allowed lateness as a business tradeoff between lost revenue and held-open window memory, and route anything past it to a dead-letter topic for reconciliation.

Pick Spark Structured Streaming with a 10-second trigger for the scoring feed if the SLA tolerates it and the team already runs Spark, and reach for Flink only when per-event latency or merging session windows demand its state model.

Close on failure handling: at-least-once with idempotent upsert sinks, enable_auto_commit=False with manual offset commit after the sink write, and Flink checkpoints whose duration stays well under the interval so checkpointing does not become its own backpressure.

KEY TAKEAWAYS

Open with "what is the latency SLA?" instead of "do we want real-time," because a number eliminates architectures and a wish does not; volume alone never decides this, since a 10TB nightly load is fine as batch and a 100-event-per-second fraud stream is not.

Name Lambda architecture explicitly, then flag dual maintenance before you are asked: every business logic change ships to both layers, and teams that ignore that end up with a speed layer whose numbers nobody trusts.

Convergence requires the speed layer to emit deltas rather than recomputed totals so the batch layer can cleanly supersede it, which is what the COALESCE plus CASE serving-layer merge expresses; sharing processing code across both layers is what prevents permanent drift.

A watermark is the system's estimate of how far behind the newest unseen event is, and allowed lateness is a business tradeoff: 30 minutes of held-open windows costs memory, while a closed window permanently loses the $500 transaction from 2:45 PM that arrived at 3:20 PM.

Session windows close on an inactivity gap, so a late event inside that gap can extend the session and merge two previously separate sessions, which is the detail that signals production experience.

At-least-once with idempotent sinks is the production default, not exactly-once: upserts and dedup keys make reprocessing harmless, and Kafka's auto-commit must be turned off with enable_auto_commit=False so offsets commit only after the sink write succeeds.

The architecture decision that defines your pipeline

Category: Pipeline Architecture
Difficulty: intermediate
Duration: 35 minutes
Challenges: 2 hands-on challenges

Topics covered: When Would You Use Streaming?, Design for Both Batch and Stream, What About Late Data?, Micro-batch or True Streaming?, How Do You Handle Failures?

Lesson Sections

When Would You Use Streaming? (concepts: Batch vs Streaming)
The Decision Framework The interviewer wants to hear a number, not a vibe. Your first sentence should be: "What's the latency SLA?" Not "do we want real-time" - that's a wish, not a requirement. If they say "daily is fine," you just eliminated streaming from the conversation. That's a senior signal - you saved six months of unnecessary infrastructure. There are exactly four factors that drive the batch-vs-stream decision. Your answer should walk through all four. Miss any one of them and the int
Design for Both Batch and Stream (concepts: Lambda Architecture)
Lambda Architecture in Practice Here's the reality the interviewer is testing: production data platforms almost always need both batch and streaming. Batch gives you correctness. Streaming gives you speed. The question is how you merge them without creating a maintenance nightmare. Your answer should acknowledge both sides and then explain the merge strategy. The Lambda architecture formalized this pattern: a speed layer (streaming) serves approximate results immediately, while a batch layer rep
What About Late Data? (concepts: Late-Arriving Data)
Watermarks, Windows, Late Arrivals This is the #1 follow-up question after you propose a streaming architecture. Interviewers commonly ask some version of: "A user clicks at 11:59 PM but the event arrives at 12:03 AM. Which day does it belong to?" If you've already closed the daily window at midnight, you've got a problem. The interviewer wants to hear three words: watermark, allowed lateness, dead-letter. Start your answer here: a watermark is the system's estimate of "how far behind is the lat
Micro-batch or True Streaming? (concepts: Micro-Batch vs True Streaming)
Spark Structured Streaming vs Flink The interviewer is testing whether you pick a framework based on requirements or based on resume keywords. "Streaming" is a spectrum, not a binary. On one end, Spark Structured Streaming processes data in small batches (micro-batches) with a latency floor around 100 milliseconds. On the other end, Apache Flink processes each event individually. Your answer should match the framework to the SLA, not to your personal preference. The trigger interval in Spark SS
How Do You Handle Failures? (concepts: Idempotency)
Exactly-Once, Offsets, Checkpoints Every streaming system will fail. The interviewer knows this. The question isn't whether failures happen - it's whether your pipeline produces correct results when they do. Your answer framework: start with delivery guarantees, then explain offset management, then describe your idempotency strategy. Hit all three and you've covered the full rubric. The interviewer wants to hear that at-least-once is the production default. Not exactly-once. The trap is answerin