Five Years of Cron Jobs

A hard Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

hard

Seniority

L6

Problem

We have a data platform that has grown organically over five years: some pipelines run on-premises as scheduled cron jobs, others are event-driven workflows triggered by upstream system callbacks. We need to migrate everything to a single cloud-native orchestration platform without disrupting the analytics team. Design the migration architecture and the logging and observability schema that supports it.

Summary

Half the jobs run on cron. Half run on events. All of it has to move.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.