A Kafka consumer rebalance reprocesses the last batch every deploy, the Stripe webhook retries until

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A Kafka consumer rebalance reprocesses the last batch every deploy, the Stripe webhook retries until it gets a 2xx, and the SFTP partner re-uploads files when their cron retries. The current pipeline writes everything that arrives, producing duplicate rows that take weeks to find. The section's pattern: at-least-once delivery is the world; deduplicate at the sink on a stable key drawn from the message itself (Kafka producer UUID, Stripe evt_id, file natural key plus source-file id, source PK plus bookmark window). Add the dedupe key by replacing the sink with one whose name states the dedupe key and the uniqueness mechanism (UNIQUE constraint, MERGE on key, partition overwrite by key window).

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.