A global trading platform consolidates the lesson into one architecture

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A global trading platform consolidates the lesson into one architecture. Three event sources (mobile retail trades, institutional order flow, market-data feed) write to a shared Iceberg target. Schema changes flow through a registry on the producer side and a CDC allow-additive policy on the database side. The streaming pipeline serves real-time desks; a daily reconciliation pass serves audit consumers; a delta log quantifies the gap. Each source has a profile-matched watermark strategy. The schema-quarantine topic catches blocked destructive changes for human decision. Compose the system: schema registry between producers and topics with compatibility named; per-source watermark transforms named for their strategy and parameters; aggregation with event_time, lateness budget, dead-letter; CDC allow-additive policy with quarantine destination; reconciliation pass with daily cadence and atomic partition overwrite; delta-metric log; both real-time and audit consumers reading the shared Iceberg target.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.