Three Providers, One Workout

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain: Pipeline Design
Difficulty: hard
Seniority: L5
Asked in: Peloton

Problem

Workout events arrive as webhooks from three providers in three formats, and about a third of sessions are reported by two or more of them. Each event carries the session's score with raw heart rate and GPS. Design the pipeline from those webhooks to a live leaderboard and to the permissioned readers of health data.

The same ride, reported three times.

Business requirements

Finishing a workout is the user-facing reward; the leaderboard has to reflect a completed workout within roughly a minute.
When a user records the same workout on a hardware device and a wearable, the leaderboard can't show two entries for one workout.
Heart rate and GPS are PHI; they can't sit on the leaderboard's own store, only permissioned consumers may read them, and every read of those fields has to land in an audit log.

Worked solution and explanation

What this really is

This is entity resolution on a one-minute clock, with a compliance boundary running through the middle. The skill being probed: can you collapse the same workout reported by a device and a wearable into one leaderboard entry while the event is still in flight, and keep heart rate and GPS off the path everyone reads? Anyone can wire webhooks to a consumer. The trap is that a consumer which updates the leaderboard from every event double-counts the workout, and the table it writes to also holds PHI. Get that wrong and the board shows two entries for one workout, and a single direct query hands heart rate and GPS to a reader nobody logged.

The reference design

Webhooks land on a bus, one streaming path canonicalizes and dedups, then splits: heart rate and GPS are stripped before the fast-path leaderboard store, and the full record goes to a separate PHI store behind a gate that logs every read.

provider webhooks (source, API)
event bus (queue, Kafka)
canonicalizer (transform, Flink)
workout dedup (transform, Flink)
strip heart rate and GPS (transform)
leaderboard store (storage, PostgreSQL); freshness: < 1min
phi store (storage, PostgreSQL)
access policy (quality gate, custom)
audit log (storage, PostgreSQL)
leaderboard users (consumer, API)
permissioned team (consumer, API)

provider webhooks -> event bus
event bus -> canonicalizer
canonicalizer -> workout dedup
workout dedup -> strip heart rate and GPS
strip heart rate and GPS -> leaderboard store
workout dedup -> phi store
leaderboard store -> leaderboard users
phi store -> access policy
access policy -> permissioned team
access policy -> audit log

Source Layer: provider webhooks
Ingestion Layer: event bus
Processing Layer: canonicalizer, workout dedup, strip heart rate and GPS
Storage Layer: leaderboard store, phi store, access policy, audit log
Serving Layer: leaderboard users, permissioned team

Step 1: Put a bus between receipt and processing

Three providers send three formats at their own pace. Acknowledge the webhook, write to the bus, and let everything downstream run at its own speed. The queue is also the fan-in point, so a slow parser never makes a provider retry.

Step 2: Canonicalize before you dedup

You cannot compare a workout from one provider with a workout from another until both have the same fields: user_id, workout type, start and end time. Normalize first, and the dedup logic is written once instead of once per provider.

Step 3: Dedup on user and workout window in the stream

Key on user_id, workout type and overlapping time window, and keep that state in the Flink job. The second report merges into the first, with a precedence rule per field, such as preferring the hardware device for heart rate. This is also what keeps the path under a minute: no batch hop between the event and the board.

Step 4: Strip PHI on the leaderboard branch

The dedup output fans out two ways. On the leaderboard branch, a stage drops heart rate and GPS, so the leaderboard store receives user, workout and score only. The full record goes to the phi store. Because the split is a stage on the canvas, no query against the leaderboard can return PHI.

Step 5: Gate every read and log it

Reads of the phi store pass through the access policy, which admits only permissioned consumers. That same node writes each read to the audit log. If the log is a side effect of the gate, no read can skip it.

One shared table: The leaderboard reads the same rows that hold heart rate and GPS. Any dashboard query can see PHI, and there is no single place to log reads.
Split stores: The leaderboard store is PHI-free, so its readers need no special rules. All PHI access funnels through one gate, which makes the audit answer trivial.

Common follow-up questions

The wearable syncs ten minutes after the hardware device. What happens to the leaderboard? (Tests late data: the merge updates the existing entry by key instead of adding a second one, and the window must be long enough to catch the late report.)
A coach is removed from a user's permissions. How does the design show that? (Tests the access policy as the single enforcement point, and the audit log as the evidence of what was read before the change.)
Where would you add dead-letter queues, and what would land in them? (Tests whether the candidate can extend the design for unparseable payloads and unresolvable dedup conflicts without changing the core path.)