The Bad Row That Broke the Dashboard
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- L6
Problem
The application event stream carries about 50,000 events per second at peak from 40 producer services, each publishing against a registered schema contract, and every event lands in Snowflake for analytics. Design the streaming pipeline that enforces those contracts at the source.
Bad records cannot reach the warehouse.
Business requirements
- Past releases let bad records into production tables and broke dashboards; validation has to happen before anything lands.
- Records that fail validation must be held aside for engineering to fix and replay, not silently dropped.
- Streaming events have to reach the warehouse within about a minute of arrival, with the quality decision made in the same window.
- Product teams reshape event fields about every week; a producer change can't crash the pipeline or corrupt downstream tables.
Worked solution and explanation
Why this problem exists in real interviews
Streaming quality enforcement with four properties that have to fit together: validation before promotion, recoverable rejection so engineering can replay, sub-minute end-to-end, and a producer contract that catches schema changes upstream. The trap is letting bad records through 'just to keep the pipeline moving' or letting validation block all events on a single bad row.
The default reach is a streaming load that writes everything and 'fixes' bad rows in a downstream cleanup. The first malformed event corrupts a dashboard; the team adds a parser fix; another field changes a week later and the same thing happens. Rejected records get logged to a file nobody reads. Producers ship a breaking schema change because the wiki page about discipline didn't catch it.
Walk the requirements
Step 1: Validate before promotion; bad records never reach the warehouse
An in-stream quality gate runs schema, type, and business-rule checks on each record before the warehouse load. Records that pass write to the production table; records that fail route to a rejection store with the failure reason. Dashboards read only from validated rows. Without a quality-check tier the validation has no place to live; without a streaming tier the validation can't keep up with sub-minute events.
Step 2: Rejected records stay recoverable for replay
Each rejection writes the original payload and the failure reason to a rejection store keyed on event id and timestamp. Engineering queries the store to find rejected records, traces the root cause (parser bug, producer change, business-rule mismatch), fixes upstream, and replays the affected events back through the pipeline. Logging rejections to a file is the version where they disappear into ops; the rejection store is what makes recovery a workflow.
Step 3: Sub-minute end-to-end including validation
Events flow through a streaming consumer, the validator, and into the warehouse within the minute. The validator is part of the streaming path, not a downstream batch; the warehouse load reads only validated rows. A 'load first, validate later' design is the version where bad data is briefly in production and downstream consumers see it. In-stream validation is what closes the gap.
Step 4: Schema contract at the bus catches producer changes at publish
Product teams add and remove fields weekly. The bus's schema-contract layer rejects publishes that don't conform to the contract; producers find out at publish time, not when consumers crash at midnight. An additive change (new optional field) is allowed through compatibility rules; a breaking change is rejected. Without the contract every producer release is a coordination tax on every consumer.
The shape that fits
- producers (source, Kafka)
- schema_contract (quality gate, custom)
- event_bus (queue, Kafka)
- streaming_validator (transform, Flink)
- quality_gate (quality gate, Great Expectations)
- rejection_store (storage, object storage)
- analytics_warehouse (storage, Snowflake)
- analytics_team (consumer, dashboard)
- engineering_replay (consumer, API)
- producers -> schema_contract
- schema_contract -> event_bus
- event_bus -> streaming_validator
- streaming_validator -> quality_gate
- quality_gate -> analytics_warehouse
- quality_gate -> rejection_store
- analytics_warehouse -> analytics_team
- rejection_store -> engineering_replay
- Producers + contract: producers, schema_contract
- Bus + validator: event_bus, streaming_validator, quality_gate
- Warehouse + rejection: analytics_warehouse, rejection_store
- Consumers: analytics_team, engineering_replay
Common follow-up questions
- A new business rule has to validate one new field; the rest of the pipeline can't change. What in this design lets the rule ship safely? (Tests whether the candidate sees the quality gate as the extension point: a new validation rule deploys to the gate, runs against new and recent records, and the rejection store flags any failures. The schema contract, the streaming consumer, and the warehouse don't change. The rule can also run in shadow mode (logging without rejecting) before going live.)
- Engineering replays a batch of rejected records after fixing the root cause. What does the design do to ensure the warehouse doesn't end up with duplicates or stale rows? (Tests whether the candidate sees that the warehouse's load is idempotent on event id; the replay re-emits records and the upsert merges them in cleanly. The rejection store records the resolution so the same record isn't replayed twice. The dedup contract is at the warehouse, not 'whatever the replay tool remembers.')