A Kafka consumer rebalance reprocesses the last batch every deploy, the Stripe webhook retries until
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
Interview Prompt
A Kafka consumer rebalance reprocesses the last batch every deploy, the Stripe webhook retries until it gets a 2xx, and the SFTP partner re-uploads files when their cron retries. The current pipeline writes everything that arrives, producing duplicate rows that take weeks to find. The section's pattern: at-least-once delivery is the world; deduplicate at the sink on a stable key drawn from the message itself (Kafka producer UUID, Stripe evt_id, file natural key plus source-file id, source PK plus bookmark window). Add the dedupe key by replacing the sink with one whose name states the dedupe key and the uniqueness mechanism (UNIQUE constraint, MERGE on key, partition overwrite by key window).
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision