Data Pipeline Architecture Patterns for Interviews

Runs batch and streaming in parallel. The batch layer processes historical data for accuracy. The speed layer processes real-time data for low latency. A serving layer merges both views. Batch is your source of truth; streaming gives you approximate real-time numbers until batch catches up. Strengths: accurate historical data, low-latency approximations. Weaknesses: two codebases, complex merge logic. Use when you need both real-time dashboards and accurate historical reports (e.g., ad click analytics where the dashboard shows live counts but billing uses batch-computed totals). Interview angle: 'why not just streaming?' Streaming gives approximate results due to late events, out-of-order events, incomplete windows. Batch gives exact results. Lambda gives both.

Streaming only. All data is a stream of events. No separate batch layer. Historical reprocessing happens by replaying the event stream from the beginning (or a checkpoint). The event log (Kafka, Kinesis) is the single source of truth. Strengths: single codebase, no merge logic, natural fit for event-sourced systems. Weaknesses: reprocessing the entire stream is expensive at scale, not all problems fit the event-streaming model, requires a durable replayable event log. Use when the use case is naturally event-driven and you can afford to reprocess for corrections. Interview angle: Kappa works when your event log is the source of truth and reprocessing is feasible. It doesn't work when batch sources (database snapshots, file exports) are your primary input.

Components communicate through events (messages). A producer publishes an event; one or more consumers react. Decouples producers from consumers: the producer doesn't know (or care) who processes the event. Message brokers (Kafka, RabbitMQ, Pub/Sub) sit in between. Strengths: loose coupling, easy to add consumers without modifying producers, natural backpressure. Weaknesses: debugging is harder (events flow through multiple systems), ordering guarantees vary by broker and config, at-least-once delivery means consumers must handle duplicates. Use for microservices architectures, real-time data platforms, any system where multiple teams produce and consume independently. Interview angle: discuss event ordering, at-least-once vs exactly-once, and how you handle duplicates.

Traditional scheduled pipelines. An orchestrator (Airflow, Dagster, Prefect) triggers jobs on a schedule or in response to data availability. Each job extracts, transforms, loads. The most common architecture in data warehousing. Strengths: simple mental model (jobs run on a schedule), mature tooling (Airflow has been production-tested for a decade), easy to reason about freshness. Weaknesses: latency is bounded by schedule interval, failures require manual intervention or complex retry logic, scaling individual jobs is harder than scaling event consumers. Use for batch analytics, data warehousing, anywhere hourly or daily freshness is acceptable. Interview angle: discuss orchestration (DAGs, dependencies, retries), idempotency, monitoring.

Lambda or Kappa decision. Start with requirements: what latency does the dashboard need? What accuracy does the historical report need? If the dashboard tolerates approximate counts, single streaming path (Kappa). If billing depends on exact counts, Lambda with a batch correction layer.

Phased, not big-bang. Phase 1: add monitoring to existing. Phase 2: decompose into modular jobs (one table per job). Phase 3: introduce orchestration (Airflow). Phase 4: migrate to a modern storage layer (on-prem to cloud). Each phase delivers value independently.

1B events/day = ~11,500/sec. Kafka for ingestion (partitioned by event key). Flink or Spark Structured Streaming for processing. Output to a columnar store (Parquet on S3 or BigQuery). Monitor lag to hit the 5-minute SLA. Discuss what happens when throughput exceeds processing capacity.

Batch: process all data at once on a schedule (hourly, daily). Micro-batch: process in small intervals (30 seconds to 5 minutes), implemented by Spark Structured Streaming. True streaming: process each event as it arrives, implemented by Flink and Kafka Streams. Batch has highest latency, simplest implementation. True streaming has lowest latency, most complex state management. Micro-batch is the practical middle ground.

Step 1: assess impact (which tables, which consumers). Step 2: disable the pipeline to prevent more bad data. Step 3: identify root cause. Step 4: fix and validate. Step 5: backfill the 3 affected days (pipeline must be idempotent). Step 6: add monitoring to catch this earlier.

Managed (Fivetran, Stitch) for standard SaaS connectors. Custom for proprietary data sources, complex transformation logic, or performance requirements managed services can't meet. The tiebreaker is often the team: do you have engineers who can maintain custom code?

True exactly-once is an illusion; it's achieved through at-least-once delivery plus idempotent sinks. Use Kafka transactions for produce-and-commit atomicity. Use Flink checkpointing for stateful processing. Use MERGE or upsert at the sink for idempotent writes.