How Data Moves

Nail the batch vs streaming question and defend your choice

Nail the batch vs streaming question and defend your choice

Category

Pipeline Architecture

Difficulty

beginner

Duration

20 minutes

Challenges

0 hands-on challenges

Topics covered: Batch Processing, Stream Processing, File Ingestion, API Ingestion, Batch vs Streaming

Lesson Sections

1. Batch Processing (concepts: paBatchProcessing)

   What They Want to Hear 'Batch processing collects data over a period and processes it all at once on a schedule. I would use it here because the consumers check dashboards once a day, so paying for real-time freshness would be waste.' That is the answer. Batch = scheduled, chunked, predictable. Say it in one sentence, then prove you understand the tradeoffs.

2. Stream Processing (concepts: paStreamProcessing)

   What They Want to Hear 'Streaming processes each event as it arrives, continuously. I would use it when stale data directly costs money or breaks the user experience, like fraud detection or live driver tracking.' Then name the tools: Kafka is the message broker that holds events. Flink or Spark Streaming is the processor that acts on them. You do not need to know how to configure these tools. You need to know what role each plays. The Tools to Name-Drop

3. File Ingestion (concepts: paFileIngestion)

   What They Want to Hear 'Files are the most common ingestion method. A vendor drops a CSV on S3, an event notification triggers the pipeline, and we validate before ingesting.' That is the baseline answer. Then show depth by naming formats and their tradeoffs: CSV is universal but slow at scale. Parquet is columnar, compressed, and 10-30x faster for analytics. JSON is flexible but verbose. Format Cheat Sheet

4. API Ingestion (concepts: paApiIngestion)

   What They Want to Hear 'We use pull-based REST APIs for scheduled data extraction and push-based webhooks for near-real-time event delivery.' Then immediately mention the two things that separate a production answer from a toy answer: rate limits and idempotency.

5. Batch vs Streaming (concepts: paBatchVsStreaming)

   The #1 Pipeline Interview Question This question is designed to test your judgment, not your knowledge. The interviewer describes a scenario and wants to see you reason through the decision, not recite definitions. Here is the framework that works every time: Step 1: Ask 'If this data is 1 hour old, does anyone lose money or make a bad decision?' If no, batch. Step 2: If yes, ask 'Does a 5-minute delay cause the same problem?' If 5 minutes is fine, micro-batch. If sub-minute matters, true stream