A new pipeline does not need fifty monitors

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A new pipeline does not need fifty monitors. The section names three day-one monitors that catch most failures in the first month: did it run (no run record exists for today), did it succeed (run exists but exited non-zero), and was the output the right size (run succeeded but row count fell outside the expected band). Together they catch scheduler outages, code errors, source schema changes, silent filters, and partial extracts. Choose the day-one monitors by adding three monitor transforms downstream of the orchestrator and the destination, one per question, each with the metric source and a paging destination wired.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.