A pipeline consumes from a Kafka click_events topic

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

A pipeline consumes from a Kafka click_events topic. The current consumer has two cardinal sins: it commits the offset before writing to the raw zone (a crash between commit and write loses the message silently), and a webhook handler from a SaaS partner writes directly to the same downstream without a buffer (peak load knocks the consumer over). The section's rules: write first, commit after; route webhook events through a queue. Place the consumer by replacing the Kafka consumer transform with one whose name states 'write-first, commit-after' and adding a queue between the webhook source and the writer so the webhook handler returns 2xx as soon as the queue accepts the event.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.