The Migration That Cannot Break Morning
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L5
- Asked in
- dbt Labs
Problem
The 6am reports cannot miss a single morning while we move 60 production pipelines from our on-premises platform to a cloud warehouse before the hardware lease ends in nine months. Design the architecture that carries the migration itself, from the old and new pipelines through to the reports.
It all works today. Moving it without losing a single report is the hard part.
Business requirements
- During the migration the existing on-prem pipelines must keep running and producing output alongside the new cloud pipelines, so nothing depends on the cloud side being correct before it is proven.
- Before any pipeline is cut over, the on-prem and cloud outputs must be compared and reconciled; consumers only move to the cloud side once the two agree.
- If a migrated pipeline misbehaves, that one pipeline has to be pointed back to its on-prem output within hours without affecting any others; the design needs a per-pipeline switch, not an all-or-nothing cutover.
Worked solution and explanation
Why this problem exists in real interviews
Migrating 60 production pipelines to the cloud without missing a 6am business-report SLA, with inter-DAG dependencies and per-pipeline rollback. The trap is migrating an upstream before its downstream and breaking consumers, or scheduling high-risk changes mid-week.
The default reach is to migrate pipeline by pipeline as the team gets to them. The first time an upstream is migrated before its downstream, the downstream's input format changes and the morning report breaks. A bad cutover happens midweek and rolling back takes most of a day; a different pipeline's morning is missed. Some pipelines have no parallel run because the team trusts the migration after spot checks.
Walk the requirements
Step 1: Morning reports run through the migration window
The orchestrator schedules each pipeline on whichever side is authoritative on a given day with one 6am SLA view. Sensors fire ahead of the deadline if any side is at risk; on-call has hours to recover. High-risk changes (cutover, rollback) are restricted to weekends so a Sunday-night issue doesn't take Monday's reports out. Without the orchestration layer there's nothing watching the deadline across both sides.
Step 2: Migrate in dependency order; downstreams never miss their inputs
Many DAGs read outputs from other DAGs. The migration plan walks the dependency graph and migrates upstreams before downstreams; a downstream's input shape doesn't change until the upstream's cutover is complete. A 'whichever pipeline is easiest first' approach is the version where a downstream's input changes mid-week and the morning report breaks; dependency-ordered cutover is the contract that prevents it.
Step 3: Per-pipeline rollback so a misbehaving pipeline reverts alone
When a migrated pipeline misbehaves after cutover, the per-pipeline switch flips the canonical pointer for that pipeline back to on-prem. The other migrated pipelines stay on cloud. The rollback is hours, not days, because it's a routing change, not a re-migration. A 'big bang rollback' is the version where one bad pipeline drags the rest back; per-pipeline rollback is what isolates the failure.
The shape that fits
- orchestrator (transform, Airflow)
- on_prem_pipelines (transform, SQL)
- cloud_pipelines (transform, Spark)
- on_prem_warehouse (storage, PostgreSQL)
- cloud_warehouse (storage, warehouse)
- parity_gate (quality gate, Great Expectations)
- per_pipeline_switch (transform, SQL)
- morning_reports (consumer, dashboard)
- orchestrator -> on_prem_pipelines
- orchestrator -> cloud_pipelines
- on_prem_pipelines -> on_prem_warehouse
- cloud_pipelines -> cloud_warehouse
- on_prem_warehouse -> parity_gate
- cloud_warehouse -> parity_gate
- parity_gate -> per_pipeline_switch
- per_pipeline_switch -> morning_reports
Common follow-up questions
- Two pipelines depend on a third that's still on-prem during their cloud cutover. What does this design do? (Tests whether the candidate sees the orchestrator's DAG spanning both sides; the cloud pipelines read the on-prem upstream through the switch until the upstream is also migrated. The switch resolves cross-side reads; the dependency order means the upstream cuts over first.)
- A migrated pipeline's parity diff has been clean for two weeks but spikes once on a quarter-end run. How does this design respond? (Tests whether the candidate sees the parity check as recurring: the diff spike halts cutover or triggers rollback if already cut over, the team investigates whether the spike is a legacy bug or a new bug, and the check stays open until the discrepancy is explained or accepted.)