Pipeline Architecture Interview Practice

This is the most common opening question in a pipeline architecture interview. The interviewer describes a data pipeline requirement and asks whether you would use batch or streaming. The answer is almost never purely one or the other. Most production systems use both, and the interviewer wants to hear you reason about when each approach fits. Key questions: A retailer needs real-time inventory counts across 5,000 stores. Batch or stream?; An ad platform needs to attribute clicks to impressions within a 30-minute window. How do you design it?; You have a daily financial report that must be exactly correct. Can you use streaming?; Your batch pipeline takes 6 hours. The business wants results every 15 minutes. What are your options?. Deep dive: Strong candidates do not just say 'streaming for real-time, batch for everything else.' They talk about latency requirements (does 'real time' mean 100ms or 15 minutes?), correctness guarantees (exactly-once semantics add complexity), operational cost (streaming infrastructure costs 3 to 10x more than batch for the same throughput), and team expertise (a team that has never run Kafka should not start with streaming for a critical pipeline). The interviewer probes each of these dimensions.

Pipelines fail. Sources go down. Files arrive late. Schemas change without warning. Containers run out of memory. The reliability section tests whether you can design systems that handle failure gracefully. This is often the section that separates senior candidates from everyone else, because it requires production experience that you cannot fake. Key questions: Your source API starts returning 500 errors halfway through an extraction. What happens to the data you already pulled?; A file that normally arrives at 6am has not arrived by 8am. How does your pipeline handle this?; A pipeline writes to a table, then fails during a downstream step. How do you prevent duplicate data on retry?; Your pipeline processes 100 files per batch. File #47 has a corrupt record. What is your strategy?. Deep dive: Interviewers at Google, Amazon, and Netflix weight reliability heavily. They have all dealt with pipelines that corrupted production data because a retry wrote duplicates, or that silently dropped records because error handling was too aggressive. They want to hear you talk about checkpointing (saving progress so retries do not start from scratch), idempotent writes (running the same pipeline twice produces the same result), dead letter queues (parking bad records instead of blocking the pipeline), and alerting (knowing within minutes when something fails, not discovering it when a dashboard is empty).

Orchestration is how you coordinate the steps of a pipeline: what runs when, what depends on what, how you handle retries, and how you manage state across tasks. Interviewers test whether you understand DAG design, dependency management, and the tradeoffs between different orchestration tools. Key questions: You have 12 pipeline steps with complex dependencies. How do you model the DAG?; Task A takes 10 minutes. Task B depends on A and takes 2 hours. Task C depends on B. The SLA is 3 hours. Where is the risk?; Your Airflow DAG has 200 tasks. It takes 45 minutes just to schedule them. What do you do?; Two pipelines need to write to the same table. How do you prevent conflicts?. Deep dive: Candidates who have only used cron jobs struggle here. Interviewers expect you to know at least one orchestration tool (Airflow, Dagster, Prefect, or equivalent) and to understand its strengths and limitations. They will ask about task-level retries (not just DAG-level), backfill support (can you rerun last Tuesday's data without rerunning the whole week?), and monitoring (how do you know when a task is slower than usual?). Strong candidates also mention the distinction between data-aware and schedule-aware orchestration.

Source schemas change. A column gets renamed. A new field appears. A field that was required becomes optional. Schema evolution tests whether your pipeline handles these changes without breaking, and whether your downstream consumers can adapt. This topic is especially important at companies integrating data from multiple source systems. Key questions: Your upstream team adds a new column to the source table. Does your pipeline break?; A column changes from integer to string in the source. How does your pipeline detect and handle this?; You use Avro schemas with a schema registry. A producer publishes a breaking change. What happens?; Your warehouse has 50 consumers reading from a table you own. You need to rename a column. What is the migration plan?. Deep dive: Schema evolution separates engineers who have maintained pipelines for years from those who have only built them. When you have been woken up at 2am because an upstream team added a column that broke your Spark job, you develop strong opinions about schema contracts, compatibility checks, and versioning strategies. Interviewers look for this experience. They want to hear about forward compatibility (can old consumers read new data?), backward compatibility (can new consumers read old data?), and the specific tools you would use to enforce schema contracts (schema registries, contract tests, or validation layers).

Your pipeline works at 1GB per day. The business grows to 1TB per day. What breaks? Scaling questions test whether you understand the bottlenecks in a data pipeline and can redesign components to handle 100x or 1000x growth. The answer is never just 'add more machines.' Key questions: Your pipeline processes 10 million events per day. Growth projections say 1 billion in 18 months. What changes?; Your single-threaded Python ETL takes 4 hours. How do you parallelize it?; You are loading data into a warehouse. At current growth, you will exceed the table size limit in 6 months. Options?; Your streaming pipeline has a consumer lag of 2 hours. How do you diagnose and fix it?. Deep dive: Interviewers probe three layers: compute (do you need more workers, bigger machines, or a different processing framework?), storage (do you need partitioning, compaction, or a different storage format?), and network (is the bottleneck data transfer between systems?). Strong candidates quantify: 'At 1 billion events per day, that is ~12,000 events per second. A single Kafka consumer can handle 50,000 messages per second, so one consumer is enough for ingestion. The bottleneck is the transformation step, which currently processes 200 events per second. We need to parallelize by partition key.'

Do you ask about latency requirements, data volume, correctness guarantees, and team constraints before proposing a design? Or do you jump straight to a solution? Engineers who ask clarifying questions consistently score higher because their designs fit the actual problem, not an assumed one.

Are your technology and pattern choices appropriate for the requirements? Did you choose batch because the SLA allows it and the team knows Airflow? Did you add a dead letter queue because the source data has known quality issues? Each decision should tie back to a requirement or constraint you identified.

Every architecture decision has tradeoffs. Streaming gives lower latency but costs more and adds complexity. Denormalization speeds reads but complicates updates. Interviewers test whether you can identify these tradeoffs and explain why you chose one side for this specific scenario. Saying 'it depends' without saying what it depends on is not enough.

Does your design handle failure? What happens when the source goes down? When a transformation crashes? When a write partially succeeds? Candidates who address failure proactively (before the interviewer asks) get significantly higher scores than those who only address it when prompted.

Can you explain your design clearly to a technical audience? Do you structure your explanation logically (requirements first, then architecture, then tradeoffs)? Do you use visual aids (even verbal descriptions like 'the data flows from source to staging to warehouse')? Clear communication is scored separately from technical correctness.