Loading...
Streaming CDC into Delta Lake with UPSERT
A hard Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- staff
Problem
Our OLTP database is under constant write pressure and we can't run analytics queries against it directly. We want to replicate it continuously into a Delta lake so analysts can query it without impacting production. The data changes constantly and our analysts need it to be current within minutes. Design the streaming pipeline.
Practice This Problem
Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it instantly.