# Three Providers, One Workout > The same ride, reported three times. Canonical URL: Domain: Pipeline Design · Difficulty: hard · Seniority: L5 · Asked in: Peloton ## Problem Workout events arrive as webhooks from three providers in three formats, and about a third of sessions are reported by two or more of them. Each event carries the session's score with raw heart rate and GPS. Design the pipeline from those webhooks to a live leaderboard and to the permissioned readers of health data. ## Business requirements - Finishing a workout is the user-facing reward; the leaderboard has to reflect a completed workout within roughly a minute. - When a user records the same workout on a hardware device and a wearable, the leaderboard can't show two entries for one workout. - Heart rate and GPS are PHI; they can't sit on the leaderboard's own store, only permissioned consumers may read them, and every read of those fields has to land in an audit log. ## Worked solution and explanation ### What this really is This is entity resolution on a one-minute clock, with a compliance boundary running through the middle. The skill being probed: can you collapse the same workout reported by a device and a wearable into one leaderboard entry while the event is still in flight, and keep heart rate and GPS off the path everyone reads? Anyone can wire webhooks to a consumer. The trap is that a consumer which updates the leaderboard from every event double-counts the workout, and the table it writes to also holds PHI. Get that wrong and the board shows two entries for one workout, and a single direct query hands heart rate and GPS to a reader nobody logged. > **Dedup after the leaderboard write is too late** > > Candidates often write every event to the leaderboard store and dedup in a nightly job. The user sees two entries for the first day, and the PHI columns already sit in a table with the wrong readers. Collapse the workout before anything is written, and split the fields at the same point. ### The reference design **Pipeline stages** - provider webhooks (source, API) - event bus (queue, Kafka) - canonicalizer (transform, Flink) - workout dedup (transform, Flink) - strip heart rate and GPS (transform) - leaderboard store (storage, PostgreSQL); freshness: < 1min - phi store (storage, PostgreSQL) - access policy (quality gate, custom) - audit log (storage, PostgreSQL) - leaderboard users (consumer, API) - permissioned team (consumer, API) **Data flow** - provider webhooks -> event bus - event bus -> canonicalizer - canonicalizer -> workout dedup - workout dedup -> strip heart rate and GPS - strip heart rate and GPS -> leaderboard store - workout dedup -> phi store - leaderboard store -> leaderboard users - phi store -> access policy - access policy -> permissioned team - access policy -> audit log **Layers** - Source Layer: provider webhooks - Ingestion Layer: event bus - Processing Layer: canonicalizer, workout dedup, strip heart rate and GPS - Storage Layer: leaderboard store, phi store, access policy, audit log - Serving Layer: leaderboard users, permissioned team *Webhooks land on a bus, one streaming path canonicalizes and dedups, then splits: heart rate and GPS are stripped before the fast-path leaderboard store, and the full record goes to a separate PHI store behind a gate that logs every read.* #### Step 1: Put a bus between receipt and processing Three providers send three formats at their own pace. Acknowledge the webhook, write to the bus, and let everything downstream run at its own speed. The queue is also the fan-in point, so a slow parser never makes a provider retry. #### Step 2: Canonicalize before you dedup You cannot compare a workout from one provider with a workout from another until both have the same fields: `user_id`, workout type, start and end time. Normalize first, and the dedup logic is written once instead of once per provider. #### Step 3: Dedup on user and workout window in the stream Key on `user_id`, workout type and overlapping time window, and keep that state in the Flink job. The second report merges into the first, with a precedence rule per field, such as preferring the hardware device for heart rate. This is also what keeps the path under a minute: no batch hop between the event and the board. #### Step 4: Strip PHI on the leaderboard branch The dedup output fans out two ways. On the leaderboard branch, a stage drops heart rate and GPS, so the leaderboard store receives user, workout and score only. The full record goes to the `phi store`. Because the split is a stage on the canvas, no query against the leaderboard can return PHI. #### Step 5: Gate every read and log it Reads of the `phi store` pass through the access policy, which admits only permissioned consumers. That same node writes each read to the audit log. If the log is a side effect of the gate, no read can skip it. **One shared table** The leaderboard reads the same rows that hold heart rate and GPS. Any dashboard query can see PHI, and there is no single place to log reads. **Split stores** The leaderboard store is PHI-free, so its readers need no special rules. All PHI access funnels through one gate, which makes the audit answer trivial. > **Every box answers a requirement** > > The tell is a design where you can point at each node and name the requirement it serves. Dead-letter queues, retries and a separate dedup cache are reasonable in production, but you should offer them as extensions, not draw them unprompted. ## Common follow-up questions - The wearable syncs ten minutes after the hardware device. What happens to the leaderboard? _(Tests late data: the merge updates the existing entry by key instead of adding a second one, and the window must be long enough to catch the late report.)_ - A coach is removed from a user's permissions. How does the design show that? _(Tests the access policy as the single enforcement point, and the audit log as the evidence of what was read before the change.)_ - Where would you add dead-letter queues, and what would land in them? _(Tests whether the candidate can extend the design for unparseable payloads and unresolvable dedup conflicts without changing the core path.)_ ## Related - [All practice problems](https://datadriven.io/problems) - [Mock interview mode](https://datadriven.io/interview/three_providers_one_workout) - [System Design Interview Questions](https://datadriven.io/data-engineering-system-design) - [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep) - [Daily Challenge](https://datadriven.io/daily) --- Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.