# A pipeline consumes from a Kafka click_events topic

Canonical URL: <https://datadriven.io/problems/a-pipeline-consumes-from-a-kafka-clickevents-topic-the-cur-9e574207>

Domain: Pipeline Design · Difficulty: medium

## Problem

A pipeline consumes from a Kafka click_events topic. The current consumer has two cardinal sins: it commits the offset before writing to the raw zone (a crash between commit and write loses the message silently), and a webhook handler from a SaaS partner writes directly to the same downstream without a buffer (peak load knocks the consumer over). The section's rules: write first, commit after; route webhook events through a queue. Place the consumer by replacing the Kafka consumer transform with one whose name states 'write-first, commit-after' and adding a queue between the webhook source and the writer so the webhook handler returns 2xx as soon as the queue accepts the event.

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/a-pipeline-consumes-from-a-kafka-clickevents-topic-the-cur-9e574207)
- [System Design Interview Questions](https://datadriven.io/data-engineering-system-design)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). 100% free data engineering interview prep. Live code execution against Postgres 16, Python 3.11, and Spark sandboxes. No paywall, no premium tier, no signup gate.