Every Device Has Its Own Dialect

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
medium
Seniority
L5
Asked in
Peloton

Problem

Our fitness platform receives workout and health events from connected devices, a mobile app, and third-party integrations. Events arrive in multiple formats at different cadences, with different schema versions across device firmware generations. Design the ingestion pipeline that normalizes these into a unified event store.

Three sources. Three formats. Same workout.

Business requirements

Worked solution and explanation

What this problem is really about

This is a time-semantics and schema-contract problem dressed up as ingestion. Anyone can draw three sources into a lake. The trap is one canonical converter that requires every field and stamps arrival time: live workouts lag, old firmware throws, and an airplane-mode workout is filed under the day it uploaded. Get that wrong and every streak breaks for exactly the users who trained offline.

Walk the requirements

Step 1: Split cadence by consumer, not by source

Live feedback needs device events within a minute, so a Flink job off the bus feeds a Redis feature store. Mobile and partner data can wait, so a Spark job on an hourly cadence is enough. Streaming compute for the slow sources buys nothing.

Step 2: Make the schema additive and registry-backed

A schema registry sits beside both normalizers. New fields are optional, old fields never disappear, and old firmware simply yields nulls. Separate tables for old and new firmware double the work on every rollout.

Step 3: Key the store on event_time

Late uploads land in the partition for the day the workout happened, and partition_overwrite rebuilds just that day. Both normalizers write one store, so a streak query never has to know which path an event took.

The reference design

Device events take the real-time path and mobile and partner events take the hourly one. Both normalize against the same registry and land in one store keyed by `event_time`.
  • device events (source, Kafka)
  • mobile events (source, API)
  • partner events (source, API)
  • event bus (queue, Kafka)
  • firmware schema registry (transform)
  • live normalizer (transform, Flink); freshness: real-time
  • batch normalizer (transform, Spark); freshness: < 1h
  • unified event store by event time (storage, Iceberg); backfill: partition_overwrite
  • live feature store (storage, Redis)
  • live feedback app (consumer, API)
  • analytics dashboard (consumer)
  • device events -> event bus
  • mobile events -> event bus
  • partner events -> event bus
  • event bus -> live normalizer
  • event bus -> batch normalizer
  • firmware schema registry -> live normalizer
  • firmware schema registry -> batch normalizer
  • live normalizer -> live feature store
  • live normalizer -> unified event store by event time
  • batch normalizer -> unified event store by event time
  • live feature store -> live feedback app
  • unified event store by event time -> analytics dashboard
  • Sources and bus: device events, mobile events, partner events, event bus
  • Schema normalization: firmware schema registry, live normalizer, batch normalizer
  • Event store: unified event store by event time
  • Serving layer: live feature store, live feedback app, analytics dashboard
One shared bus
A single event bus gives one fan-out point and one place to replay from. The cost is that all three sources share its retention and partitioning.
Per-source lanes
A dedicated streaming lane for devices plus a scheduled merge for mobile and partner data is just as correct. What matters is that the three properties hold, not that a shared bus exists.

Common follow-up questions