A logistics ETL processes shipment events from Kafka, enriches each with a third-party route API, an
A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.
- Domain
- Pipeline Design
- Difficulty
- medium
Problem
A logistics ETL processes shipment events from Kafka, enriches each with a third-party route API, and writes to Snowflake. Four beginner-tier failure modes are present: errors are uncategorized, retries are unbounded with bare except, 200 workers retry on the same one-second clock, and malformed rows loop forever. Assemble the resilient pipeline. Replace the broken enrich transform with one whose name states the full retry policy (bounded attempts, capped exponential backoff, jitter, transient-only). Add three terminal destinations for validation failures, authentication failures, and poison pills. The Snowflake destination must remain on the production read path.
Practice This Problem
Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.