# Data Pipeline Interview Questions > End-to-end pipeline design and operations problems for data engineer interview prep. Canonical URL: Breadcrumb: [Home](https://datadriven.io/) > [Data Pipeline Interview Questions](https://datadriven.io/data-pipeline-interview-questions) ## Summary Data pipeline interview questions for data engineer roles, covering the design round and the operational round. End-to-end design across ingest, transform, and serve layers. Streaming and batch architectures. Idempotent transformation with MERGE INTO. Multi-source ingestion. Multi-region replication. The architecture rounds that data engineer L5+ loops expect. ## What this page covers Data pipeline design interview questions span ingest, transform, and serve layers end-to-end. The ingest layer captures data from sources (CDC, event streams, batch dumps, API pulls); the transform layer applies dedup, conformed-dimension joins, business rules, and aggregation; the serve layer makes the transformed data accessible (analytical warehouse for BI, online feature store for ML serving, materialized view for dashboards). Each layer has design questions that a senior data engineer is expected to answer. Ingest layer design questions. Source type: transactional database (use CDC via Debezium), event stream (use Kafka/Kinesis/Pub/Sub directly), API (use scheduled batch pulls with high-water-mark), file drop (use S3 event trigger to Spark). Throughput sizing: events per second, peak factor, bytes per event. For Kafka: 10-20 MB/sec per partition. For Kinesis: 1 MB/sec per shard. Durability: replication factor (3 for Kafka), at-least-once delivery, snapshot recovery for new sources. Schema: registry-enforced contracts, additive-only evolution, raw payload preserved in bronze for replay. Transform layer design questions. Compute engine: Spark for distributed batch, Flink for stateful streaming, dbt for in-warehouse SQL transformations, custom Python for lightweight glue. Idempotency: run_id baked into output partitions, MERGE INTO on composite natural keys, append-only with version column for slowly-changing facts. Late-arriving data: MERGE-ADD-not-REPLACE for windowed aggregations, watermark plus allowed lateness for streaming, backfill plan for batch. Dependency management: orchestrator (Airflow, Dagster) handles cross-pipeline dependencies; downstream pipelines wait on upstream completion via sensors or asset-based triggers. Serve layer design questions. Analytical warehouse for BI: Snowflake, BigQuery, Redshift, Databricks with star schemas in the gold layer. Online feature store for ML: Redis or DynamoDB for 10ms read latency with batch backfill. Materialized view for dashboards: Snowflake materialized views, BigQuery materialized views, or pre-computed gold tables refreshed by dbt. Multi-region serving: read replicas with eventual consistency, or active-active with CRDT-based conflict resolution. Caching: CDN-fronted for static, ElastiCache-fronted for dynamic. Operational concerns: SLA monitoring, query performance dashboards, cost attribution per consumer. The 45-60 minute data pipeline design round expects the data engineer to cover all three layers in the time allotted, with explicit failure-mode articulation at each. Pacing is critical: 10 minutes on the ingest layer, 15 minutes on transform, 10 minutes on serve, with the remaining 10-25 minutes for failure-mode drills and adapt-on-fly pivots. Spending 30 minutes on the ingest layer at the expense of transform and serve is a pacing failure even if the ingest design is excellent. Companies that emphasize end-to-end data pipeline design in data engineer interviews: Netflix (streaming-heavy with Iceberg and Spark), Stripe (idempotent reconciliation across all three layers), Meta (large-scale clickstream and ads attribution), Amazon (AWS-native end-to-end), Google (GCP-native with Pub/Sub plus Dataflow plus BigQuery), Uber (Kafka plus Flink plus Pinot for real-time serving plus Spark for batch). ## Frequently asked questions ### What does a data pipeline design interview round cover? End-to-end design across ingest, transform, and serve layers. 45-60 minutes. Specific scenario: 10B events per day clickstream, daily Postgres-to-Snowflake ETL, ML feature store. Senior data engineer rubrics weight idempotency at each layer, failure-mode articulation, cost reasoning, and adapt-on-fly when the interviewer flips a requirement. ### How does a data engineer pace a 45-minute pipeline design round? 10 minutes ingest, 15 minutes transform, 10 minutes serve, 10-25 minutes for failure-mode drills and pivot. Spending 30 minutes on the ingest layer at the expense of transform and serve is a pacing failure even if the ingest design is excellent. ### What is the ingest layer in a data pipeline? The layer that captures data from sources. Mechanisms: CDC via Debezium for transactional databases, Kafka/Kinesis/Pub/Sub for event streams, scheduled batch SELECT for warehouses, API pulls for SaaS sources, S3 event triggers for file drops. Sizing: throughput in events per second, peak factor, bytes per event. ### What is the transform layer in a data pipeline? The layer that applies dedup, conformed-dimension joins, business rules, and aggregation. Compute engines: Spark for distributed batch, Flink for stateful streaming, dbt for in-warehouse SQL, custom Python for lightweight glue. Idempotency design (run_id, MERGE INTO) is the senior data engineer rubric item. ### What is the serve layer in a data pipeline? The layer that makes transformed data accessible. Analytical warehouse for BI (Snowflake, BigQuery). Online feature store for ML (Redis, DynamoDB). Materialized view for dashboards. Multi-region serving for global products. Caching, SLA monitoring, and cost attribution are operational concerns. ### How does a data engineer handle multi-source ingestion? One bronze layer per source type (transactional, event stream, batch, API), with consistent metadata: load_date, source_system, ingestion_method, raw_payload. Silver layer applies conformed dimensions across sources (one dim_customer used by orders from Postgres, events from Kafka, leads from Salesforce). Gold layer presents unified analytical models. ### What is the typical failure mode in pipeline design rounds? Not articulating failure modes at every component. A high-level architecture without 'what happens when Kafka brokers die, when Spark executors OOM, when Snowflake MERGE deadlocks' falls short of the L5 rubric. The L4 candidate produces a working architecture; the L5 candidate names 3 failure modes per component proactively. ### How does a data engineer prepare for the adapt-on-fly pivot? Practice with a peer or AI mock interviewer that flips a requirement mid-round. Common flips: SLA tightens from 15 min to 1 min, volume jumps 100x, downstream BI tool cannot handle table swaps, multi-region requirement added. The L5 signal is modifying the existing design in place and articulating what changes; the L4 signal is restarting from scratch. ## How a data engineer designs an end-to-end data pipeline Seven-step framework spanning ingest, transform, and serve layers. ### Step 1: Clarify the SLA and source Throughput, latency, freshness, durability, replay window. Source type drives ingest mechanism choice. ### Step 2: Design the ingest layer CDC for transactional, Kafka for event stream, batch SELECT for warehouse-to-warehouse. Size from throughput. ### Step 3: Design the bronze raw layer Immutable, partitioned by load_date, schema-on-read for flexibility, replay-enabled. ### Step 4: Design the silver transform layer Spark or dbt. Dedup on composite key. Conformed dimensions across sources. ### Step 5: Design the gold layer Star schemas, materialized views, business-ready aggregates. ### Step 6: Design the serve layer Warehouse for BI, online feature store for ML, materialized view for dashboard. Caching and SLA monitoring. ### Step 7: Drill failure modes and adapt 3 failure modes per component. Handle the mid-round pivot by modifying in place. ## Related practice catalogs - [Pipeline architecture practice problems](https://datadriven.io/pipeline-architecture-practice-problems): Rubric-scored architecture problems: ingest, transform, serve, with trade-off defense. - [Full data engineer interview question catalog](https://datadriven.io/data-engineer-interview-questions): Every domain in one catalog: SQL, Python, data modeling, pipeline design, system design. - [Data engineer system design interview prep](https://datadriven.io/system-design-interview-prep): Full design prep across multiple scenarios. - [Data engineering system design questions](https://datadriven.io/data-engineering-system-design-questions): Scenario catalog with end-to-end pipeline patterns. - [Data engineer system design problems](https://datadriven.io/data-engineer-system-design-problems): Practice problems with rubric-scored verdicts. - [Data pipeline practice problems](https://datadriven.io/data-pipeline-practice-problems): Hands-on pipeline design practice. - [Streaming system design interview questions](https://datadriven.io/streaming-system-design-interview-questions): Streaming-specific patterns. - [ETL design interview prep](https://datadriven.io/etl-design-interview-prep): ETL and ELT patterns within the pipeline. - [CDC pipeline interview questions](https://datadriven.io/cdc-pipeline-interview-questions): CDC as the ingest layer. - [Kafka system design interview questions](https://datadriven.io/kafka-system-design-interview-questions): Kafka as the streaming layer. - [Clickstream pipeline interview questions](https://datadriven.io/clickstream-pipeline-interview-questions): Clickstream-specific pipeline. --- Source: DataDriven (https://datadriven.io). 100% free data engineering interview prep. Live code execution against Postgres 16, Python 3.11, and Spark sandboxes. No paywall, no premium tier, no signup gate.