A Million Cars Phoning Home

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L6

Problem

Design the pipeline that carries telemetry from millions of connected vehicles to two teams: safety analytics, which acts on individual braking and ADAS events, and predictive maintenance, which reads aggregated trends. Raw coordinates may be kept only for the regulatory window, while analytics reads a coarsened location.

Every vehicle is a sensor. Deploy the pipeline to catch it all.

Business requirements

• Safety analytics needs to see an event the moment it happens; predictive maintenance can wait minutes.
• Vehicle location is personal data under European law; we can't fail a GDPR audit.

Worked solution and explanation

What this really is

This is two latency budgets and one privacy boundary sharing a single Kafka feed. Anyone can draw a stream processor. What separates candidates is whether they split the feed by who needs it how fast, and whether they decide where precise coordinates are allowed to land. Run everything through one hot path and maintenance pays real-time prices for trends it reads every fifteen minutes. Let raw lat/long reach the warehouse and every analyst query is a GDPR finding.

Walk the requirements

Step 1: Coarsen on the way in

location_coarsening is a Flink job right after vehicle_telemetry_raw that swaps raw coordinates for a coarse grid cell. It runs in the stream, not later in batch, so both teams downstream only ever see the coarse value, the safety console included.

Step 2: Fan out by latency budget

safety_events filters braking, airbag and ADAS signals into safety_store, which safety_console reads within seconds. maintenance_trend_aggregation is a Spark job on a fifteen-minute cadence that writes trends to maintenance_warehouse. Keep the hot path narrow: its cost scales with what you put on it.

Step 3: Give precise location one home with a clock on it

Raw events also land in regulatory_window_location_archive, object storage with a lifecycle rule that deletes them at the edge of the regulatory window. The only reader is audited_location_access, an API that logs every request. When an auditor asks, you answer with a retention policy and an access log.

One streaming path for everyone

Maintenance trends run on real-time compute they never asked for. Raw coordinates flow straight into the shared stores, and the privacy boundary becomes a convention people are trusted to follow.

Split by latency and by sensitivity

Safety pays for seconds and maintenance pays for minutes. Precise location has one retention-bound home, and everything analytical reads a coarsened cell.

The reference design

Common follow-up questions

• The maintenance team wants vehicle-level location traces for failure analysis. What changes? (Do they route the request through `audited_location_access` instead of adding precise coordinates to `maintenance_warehouse`?)
• Safety responders need the exact crash location. Does that break the boundary? (Do they keep exact coordinates out of `safety_store` and fetch them per incident through `audited_location_access`?)