Greenfield Build for Six Sources

A hard Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant feedback.

Domain

Pipeline Design

Difficulty

hard

Seniority

L5

Problem

We're standing up a fresh data platform on Databricks: six sources that need to be ingested, transformed through medallion layers, and served to business analysts through one consistent semantic layer. Analysts open dashboards at 8am every weekday, so every pipeline feeding them must complete by 7:30am and alert the moment a source is at risk of missing that window. Leadership currently sees a different revenue number in every meeting because each analyst writes their own KPI SQL, and finance analysts must never see raw card data from the payments feed, so every consumer has to read the metrics through a governance layer that enforces column-level access rather than off the tables directly. Design the end-to-end platform: the infrastructure-as-code config for each source, the orchestration DAG, one canonical definition per KPI in the semantic layer, and the governance checkpoint that sits between the serving layer and the dashboards to enforce column-level access.

Summary

Infrastructure as code. Meaning as a service.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and checks it automatically.