Design a Pipeline

The universal pipeline architecture question

Challenges

0 hands-on challenges

Lesson Sections

1. Decomposing the Prompt (concepts: paEltVsEtl, paDagOrchestration)

   When an interviewer says "design a pipeline to ingest clickstream data from our mobile app into our analytics warehouse," they are not asking you to start writing Spark code. They're asking: can you think in layers? The single biggest mistake candidates make is diving into implementation before establishing scope. The Five-Layer Framework Every pipeline decomposes into five layers. Naming them explicitly in the first 60 seconds of your answer signals seniority. Say: "I'll walk through this in fi

2. The Ingestion Layer (concepts: paFileIngestion, paApiIngestion, paCdc)

   When the interviewer asks how data gets into your pipeline, they are probing for a real choice, not a list. The three source patterns (file drops, API pulls, and Change Data Capture) have different reliability, latency, and cost profiles. Name which one you chose and why before they have to ask. File-Based Ingestion The simplest and most common pattern: a source system drops files (CSV, JSON, Parquet) into cloud storage (S3, GCS, ADLS). Your pipeline picks them up on a schedule. This is the defa

3. The Transformation Layer (concepts: paEltVsEtl, paMedallion, paPartitioning, paColumnarVsRow)

   The transformation layer is where the interview is won or lost. This is where interviewers spend the most time probing, because it reveals whether you understand data modeling, partitioning, and the ELT vs ETL tradeoff - the single most tested concept in pipeline interviews. ELT vs ETL: The #1 Tested Concept ETL (Extract-Transform-Load) transforms data before loading it into the warehouse. ELT (Extract-Load-Transform) loads raw data first, then transforms it inside the warehouse. This isn't ju

4. The Serving Layer (concepts: paColumnarVsRow, paPartitioning)

   The serving layer is where most candidates go thin. They spend 25 minutes on ingestion and transformation, then say 'and then analysts query it.' That's a missed opportunity. How data is consumed drives the entire upstream design - and interviewers know it. Consumer Archetypes Different consumers need different data shapes. Analysts writing SQL dashboards need pre-aggregated, denormalized gold tables with low query latency. Data scientists building ML features need wide tables with historical

5. The Meta Layer (concepts: paDagOrchestration, paMonitoring, paDataQuality)

   Orchestration, data quality, and monitoring are the meta layer - the infrastructure that makes a pipeline a pipeline instead of a script. Candidates who skip this layer cap themselves at 'hire.' Candidates who treat it as first-class get 'strong hire.' The meta layer is where you prove you've operated pipelines in production, not just built them. Orchestration: DAGs, Not Scripts A production pipeline isn't a Python script that runs on a cron job. It's a directed acyclic graph (DAG) of tasks wi