Deduplicate the Stream
A hard Spark mock interview question on DataDriven, with an AI interviewer who asks follow-up questions and a hire or no-hire decision at the end.
- Domain
- Spark
- Difficulty
- hard
- Seniority
- L5
Interview Prompt
Since the broker restart, our consumers keep receiving the same event again under a fresh `msg_id`, so the only identity that survives a replay is the producer's key in `part_key`. From `stream_msgs`, show each consumer's total deliveries next to how many unique events it actually received.
Summary
Kafka replayed 15 minutes. Your dashboard doubled.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write and run your Spark code against the tables
- Answer the interviewer's follow-up questions, then get a hire or no-hire decision