Event System for Multiple Consumers

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
hard
Seniority
L6

Problem

Every event, at a peak of 150,000 per second from about 40 producing services, has to reach six teams, from fraud to finance reporting. Design the ingestion layer and processing pipeline for the platform team's shared event system.

One event, many hungry consumers.

Business requirements

Worked solution and explanation

The trap beneath the platform

This is a multi-tenant isolation problem wearing an event-platform costume. Six teams read the same events at six speeds and deploy on their own schedules, and one producer's schema change can silently break someone else's consumer. Anyone can draw a single Kafka topic. The skill is giving each team its own consumer group, rejecting breaking schema changes at publish time, and running two freshness tiers off one bus. Get the shared consumer group wrong and one dead consumer stalls all six teams.

Walk the requirements

Step 1: Gate producers on the contract

Producers publish through the schema registry, which holds a contract per topic and rejects an incompatible change (a dropped required field, a changed type) before it reaches the bus. The producing team sees the failure at deploy time instead of three consumers crashing at midnight.

Step 2: Put one event bus behind the gate

A single Kafka-backed event bus is both the fan-out point and the retention buffer. Every team reads the same events from it, so there is no separate per-team ingestion path to keep consistent.

Step 3: Split consumers by freshness budget

The fast team runs a Flink job at real-time freshness into a PostgreSQL store. The slow team runs a Spark batch at < 1h into a warehouse. Both read the bus through their own consumer group, so neither sets the other's pace and a dead batch job never touches the stream.

Step 4: Make deletion reach every store

A deletion orchestrator on Airflow sends the delete to each team's store directly, because forgetting a customer only on the bus leaves behind the copies that matter. It tracks a confirmation per store and alerts when one is missing past the deadline, so proof of deletion is a record, not a manual hunt.

The shape that fits

The registry rejects breaking changes before the bus. Each team consumes at its own pace. The deletion orchestrator reaches every store directly.
  • producers (source, Kafka)
  • schema registry (quality gate, custom)
  • event bus (queue, Kafka)
  • fast team stream (transform, Flink); freshness: real-time
  • slow team batch (transform, Spark); freshness: < 1h
  • fast team store (storage, PostgreSQL)
  • slow team warehouse (storage, warehouse)
  • deletion orchestrator (transform, Airflow); alerting: Deletion confirmations missing past SLA
  • producers -> schema registry
  • schema registry -> event bus
  • event bus -> fast team stream
  • event bus -> slow team batch
  • fast team stream -> fast team store
  • slow team batch -> slow team warehouse
  • deletion orchestrator -> fast team store
  • deletion orchestrator -> slow team warehouse

Common follow-up questions