A 2018-era system on the canvas runs the same aggregation logic in two engines: a plain Spark nightl
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
Interview Prompt
A 2018-era system on the canvas runs the same aggregation logic in two engines: a plain Spark nightly batch transform feeding the daily executive dashboard, and a Flink streaming transform feeding the live ops dashboard. Two codebases, two failure profiles, two on-call rotations, and silent drift between the two views when one team changes the logic and the other team forgets. Apply the unified-engine framing this section just taught and collapse the two transforms into a single unified-engine transform (Spark Structured Streaming or Beam) that writes to both views; the daily view runs with trigger=once, the live view runs with trigger=processingTime='1 minute'. The application code is identical for both rhythms; only the trigger configuration differs. Remove (or replace) the plain Spark and Flink transforms; both downstream views now read from the unified pipeline. The rhythm choice becomes a config decision, not a separate codebase.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision