Six Hours to Miss a Deadline

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

We process financial data for credit risk models and regulatory reporting. Our current warehouse pipeline runs nightly full refreshes that take over six hours and frequently miss the 5am SLA. The data engineering team has been asked to redesign the pipeline using an incremental strategy, but there are concerns about correctness for slowly changing source data. Design the pipeline.

Summary

The rebuild works. It just doesn't finish in time.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.