Four Teams, One Topic, No Agreement
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L5
- Asked in
- AppFolio
Problem
Four teams publish to one shared event topic, and three streaming jobs read it, the dashboard team's real-time job among them. How would you design the pipeline so an incompatible schema change is refused when it is published, not discovered when a job crashes, and an event a job still cannot read is kept durably for replay?
Everybody is writing to it. Nobody documented it. Now production is fragile.
Business requirements
- Four producer teams deploy independently and unilateral schema changes have been breaking the consumer jobs and waking on-call.
- Three consumer jobs read from the same topic on different release cycles; if one trips on a schema issue it can't stop the others from making progress.
- The real-time dashboard team has a contracted latency on this topic; schema enforcement can't add meaningful overhead per message.
Worked solution and explanation
Why this problem exists in real interviews
This is a contract-enforcement problem dressed up as a shared-topic question. Anyone can draw one Kafka topic and three jobs; what separates candidates is where the contract is enforced. Treat schema drift as a discipline problem (more code reviews, a wiki page) and the contract lives in standups, so four independently deploying teams will keep breaking three independently deploying jobs at midnight. Put the check at publish, give each job its own progress, and give unreadable events somewhere durable to wait.
The simple answer is one Kafka topic, three consumer jobs, and a 'best practices' wiki page about schema discipline. A producer team adds a required field on Tuesday; one consumer crashes Tuesday night, and the on-call engineer rolls back the producer's change and writes a postmortem. A different producer team renames a field two weeks later; a different consumer crashes; another postmortem. Each outage spreads because the bad event keeps getting redelivered.
Walk the requirements
Step 1: Refuse incompatible schemas at publish, not downstream
The registry holds a contract per topic, and publishers serialize through it. An incompatible change (removed required field, type change, rename without an alias) is rejected at publish time. The producing team sees the error in its own deploy and either evolves the contract compatibly or reverts. Catching breaking changes in code review is the version where four producers ship and three consumers crash at midnight.
Step 2: Give each job its own progress
Each consumer is its own consumer group with its own offsets. When one job fails on an unexpected event, the topic retains messages within retention and the other jobs don't notice. Once the failing job is fixed, it resumes from where it left off. A shared consumer group across teams is the version where one team's restart blocks every other team.
Step 3: Quarantine what a job cannot read, and keep moving
When a job fails to read an event, it writes the raw event and the failure reason to a quarantine in object storage and moves on. The dashboard's job never pays for a separate validation pass: the check already happened at publish, so its hot path only deserializes against a cached schema. Triage replays from the quarantine once the reader is fixed. Letting the bad event halt the job is how a single malformed message breaks the dashboard's latency contract.
The shape that fits
- publishing_services (source, Kafka)
- publish_schema_check (quality gate, custom)
- multi_team_topic (queue, Kafka)
- dashboard_streaming_job (transform, Spark); freshness: < 1min
- lake_streaming_job (transform, Spark)
- feature_streaming_job (transform, Spark)
- dashboard_serving_db (storage, PostgreSQL)
- lake_object_store (storage, object storage)
- feature_serving_store (storage, Feast)
- replay_quarantine (storage, object storage)
- publishing_services -> publish_schema_check
- publish_schema_check -> multi_team_topic
- multi_team_topic -> dashboard_streaming_job
- multi_team_topic -> lake_streaming_job
- multi_team_topic -> feature_streaming_job
- dashboard_streaming_job -> dashboard_serving_db
- lake_streaming_job -> lake_object_store
- feature_streaming_job -> feature_serving_store
- dashboard_streaming_job -> replay_quarantine
- lake_streaming_job -> replay_quarantine
- feature_streaming_job -> replay_quarantine
Common follow-up questions
- A producer ships a 'compatible' schema change that adds an optional field; one consumer's deserializer breaks anyway. What does the design require from consumers, and where should the fix go? (Tests whether the candidate sees the contract as two-sided: the registry's compatibility rule defines what compatible means, and consumers' deserializers have to tolerate unknown fields. The fix belongs in the consumer's deserializer.)
- The quarantine is filling up because one upstream system has been emitting malformed events for hours. What does the dashboard see, what does the analytics side see, and what does on-call do? (Tests quarantine operations: good events keep flowing to the dashboard, the bad events are missing downstream and waiting in the quarantine, and on-call engages the upstream owner.)