A 2018-era system on the canvas runs the same aggregation logic in two engines: a plain Spark nightl

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A 2018-era system on the canvas runs the same aggregation logic in two engines: a plain Spark nightly batch transform feeding the daily executive dashboard, and a Flink streaming transform feeding the live ops dashboard. Two codebases, two failure profiles, two on-call rotations, and silent drift between the two views when one team changes the logic and the other team forgets. Apply the unified-engine framing this section just taught and collapse the two transforms into a single unified-engine transform (Spark Structured Streaming or Beam) that writes to both views; the daily view runs with trigger=once, the live view runs with trigger=processingTime='1 minute'. The application code is identical for both rhythms; only the trigger configuration differs. Remove (or replace) the plain Spark and Flink transforms; both downstream views now read from the unified pipeline. The rhythm choice becomes a config decision, not a separate codebase.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.