Five Years of Cron Jobs

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L6

Problem

Our data platform grew over five years into a mix of on-premises cron jobs and pipelines triggered by upstream system callbacks, and it all has to move to one cloud-native orchestration platform while the legacy runs stay live until each pipeline proves out. Design the migration architecture and the run logging and observability schema behind it.

Half the jobs run on cron. Half run on events. All of it has to move.

Business requirements

• Analytics teams rely on these pipelines daily; the migration cannot create a gap where they see stale data or missed runs.
• Audit requires that every pipeline run be queryable for years; today logs are flat text files that no one can search across.
• About a third of pipelines fire on upstream events rather than on a schedule; their behavior can't change after migration.
• For each migrated pipeline, the cloud and on-prem runs are compared during dual-run; if they diverge, on-prem stays authoritative.

Worked solution and explanation

What this really is

This is a strangler-fig migration dressed up as a logging question. Anyone can draw Airflow and an object store. What separates candidates is treating cutover as an evidence-gated decision: every pipeline dual-runs, a parity check compares cloud against on-prem, and on-prem stays authoritative until the diff holds. Skip that and three things break. Analytics becomes your test suite, the third of pipelines that fire on callbacks silently start waiting for a cron tick, and the first audit question gets answered with grep.

Walk the requirements

Step 1: Dual-run every migrating pipeline

Cloud and legacy both run for the agreed window, each writing to its own output. The old path never stops, so analytics sees no freshness gap and no missed run.

Step 2: Keep event triggers as event triggers

Route upstream callbacks onto a queue the orchestrator listens to, so each run starts when the upstream is ready. Fold them into a schedule and you add a polling delay nobody signed up for. Scheduled and event-triggered runs stay separate paths.

Step 3: Diff the outputs and let on-prem win

The parity gate compares cloud and legacy outputs run by run. Out of tolerance means legacy stays authoritative. Within tolerance for the full window flips the pointer in migration_state. No manual sign-off.

Step 4: Write a structured record for every run

Every run writes a row to pipeline_run_log: run id, pipeline, trigger type, start, end, status, input and output pointers, error context. Task detail goes to task_execution_log. The log lands in object storage as a queryable table kept for the regulatory window.

The reference design

Port and switch

Pipelines move one at a time and analytics is repointed after a spot check. Discrepancies surface a week later, reported by the analytics team. Rollback means re-enabling cron by hand.

Dual-run with parity gate

Both systems run side by side, a daily diff decides, and the flip is config. Rollback means flipping the pointer back, and the diff history explains why.

Common follow-up questions

• A pipeline's diff has held for two weeks, then spikes on a quarter-end run. What happens? (Whether the gate is continuous: the spike halts or rolls back cutover until someone decides whether legacy or cloud is wrong.)
• An auditor wants the input data of a run from two years ago. Where does the log point them? (Whether the log stores pointers rather than copies. The input data follows its own retention policy.)