45 Minutes Turned Into 3.5 Hours

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

We run a Databricks platform for bank transaction analytics, where a daily aggregation must land before the 6am risk dashboard. That job should take 45 minutes but now runs 3.5 hours and misses the deadline, and weeks of adding compute have not helped. Design the scheduled pipeline and remediation plan that gets the run back under SLA, alerts on-call before 6am if the run is at risk, and keeps the queryable output feeding the dashboard.

Summary

Spark jobs are running. Just not fast enough.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.