The Early Warning
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- mid
- Asked in
- Epic
Problem
What pipeline should a hospital's clinical data lead build so one feed of millions of vital-sign and clinical events a day, from bedside monitors and EHR systems, drives both deterioration alerts at the nursing station and the regulatory record behind next morning's compliance reports?
Business requirements
- When a patient's vitals cross a deterioration threshold, the charge nurse needs to know within seconds, not at the next shift change.
- Compliance reporting has to account for every clinical event exactly once; a duplicated or dropped event is an audit finding.
- The feed runs into the millions of events a day and spikes during rounds; the pipeline can't drop events under load.
Worked solution and explanation
Why this problem exists in real interviews
This looks like one pipeline but it is two consumers with opposite correctness budgets sharing a feed. Clinicians want a deterioration alert in seconds and will accept an approximate, best-effort signal; compliance wants every event counted exactly once for a report that runs tomorrow morning and does not care about latency at all. The trap is the single path that serves neither well: too slow to page a nurse, or too loose to survive an audit. The interviewer is watching whether you split the path by latency budget instead of defaulting to stream-everything.
The whiteboard reflex is to stream the whole feed into one table that both the alerting UI and the compliance report read. Alerts lag because the job is also doing the heavy exactly-once bookkeeping the report needs, and the report double-counts when the stream replays after a restart because nothing deduplicates on a stable id. Both stakeholders are unhappy in different directions, and one of them is a patient-safety problem.
Walk the requirements
Step 1: Buffer the ingest before you process anything
Monitors and EHRs emit in bursts, heaviest during morning rounds and code events. A durable queue decouples those producers from the processing tiers: ingest spikes turn into queue depth, not dropped events or backpressure on a bedside device. It also becomes your replay log when a downstream tier needs to be rebuilt.
Step 2: Put alerting on a streaming, sub-minute path
The deterioration detector reads the queue, evaluates each reading against thresholds, and emits to an alert destination the charge nurse actually sees, inside ten seconds. This path is allowed to be approximate (a reading briefly in flight is fine) because the cost of slowness here is measured in patient outcomes, not in a reconciliation ticket. Nothing about exactly-once belongs on this path.
Step 3: Make the warehouse load exactly-once on a stable id
Compliance reads next-day from the warehouse and must reconcile to the source event count. Each event carries a stable id from the device or EHR; the load reads the queue on its own and upserts on that id, so a restart that replays from the last checkpoint cannot double-count, and gap-free checkpointing means it cannot under-count either. This is where the exactly-once complexity lives, isolated from the latency-sensitive path and independent of whether alerting is up.
The shape that fits
- clinical_events (source, Kafka)
- deterioration_detector (transform, Flink); freshness: real-time
- nurse_alerts (consumer, alerting)
- exactly_once_loader (transform, Spark); idempotency: upsert
- clinical_warehouse (storage, warehouse)
- compliance_reporting (consumer, dashboard)
- clinical_events -> deterioration_detector
- deterioration_detector -> nurse_alerts
- clinical_events -> exactly_once_loader
- exactly_once_loader -> clinical_warehouse
- clinical_warehouse -> compliance_reporting
Common follow-up questions
- The detector goes down for ten minutes during a busy shift. What did clinicians miss, and how do you make sure it never happens silently? (Tests whether the candidate pages on detector health and treats an alerting gap as a safety incident, not just a metrics blip.)
- A device integration starts sending occasional duplicate events with the same id but a corrected value. What does the warehouse load do? (Tests the limits of dedup-on-id: an exact duplicate is absorbed, but a same-id-different-value correction needs an explicit last-write or versioning rule.)
- A misconfigured monitor starts emitting readings the detector cannot parse. Where do they go, and how does clinical review get them back? (Tests whether the candidate adds a recoverable side route so one bad reading cannot stall alerts for every other patient.)