The Fleet That Never Stops
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- L5
- Asked in
- Ecom Express
Problem
Delivery trucks stream GPS pings continuously, adding several terabytes of telemetry a day that must be retained long term. Data science pulls that history in hourly loads to train route-optimization models. Design the pipeline that feeds both the operations map and the analytics archive.
Every truck is talking. Not everyone can hear them yet.
Business requirements
- Operations watches every truck on a live map; a position has to appear there within seconds of the device emitting it.
- Trucks lose connectivity and dump buffered events when they reconnect; the history has to come out complete.
- Driver location is treated as personal data under company policy; data science can't see raw GPS coordinates.
Worked solution and explanation
This is two products sharing one firehose, dressed up as fleet tracking. The live map wants the newest ping within seconds. The archive wants every ping in the order the truck lived it, including pings that arrive hours late. Anyone can draw a stream into a map. The trap is letting one path serve both, and letting data science read the same raw coordinates. Get that wrong and a truck leaving a tunnel teleports across its route history, and privacy review finds raw driver locations in a notebook.
The shape that fits
- truck_gps (source)
- ingest_queue (queue, Kafka)
- live_stream (transform, Flink); freshness: real-time
- live_map_store (storage, PostgreSQL)
- live_map (consumer, Grafana)
- event_archive (storage, object storage)
- location_masking (transform, Spark); freshness: < 1h
- analytics_warehouse (storage, warehouse)
- data_science (consumer, Jupyter)
- truck_gps -> ingest_queue
- ingest_queue -> live_stream
- live_stream -> live_map_store
- live_map_store -> live_map
- ingest_queue -> event_archive
- event_archive -> location_masking
- location_masking -> analytics_warehouse
- analytics_warehouse -> data_science
Step 1: Put a durable queue right behind the trucks
A truck that reconnects after hours dumps its backlog in one burst. ingest_queue retains and replays, so that burst is absorbed instead of dropped, and neither consumer can slow the other down.
Step 2: Fork into a fast path and a faithful path
live_stream keeps only the newest position per truck in live_map_store, comparing event time so a late replay never overwrites a fresher ping. In parallel, event_archive lands every raw event partitioned by event time, not arrival time. That is what lets a replayed hour fall back into the hour it belongs to.
Step 3: Mask on the way to analytics, not at read time
The hourly location_masking job reads the archive and coarsens coordinates into spatial cells before anything lands in analytics_warehouse. Raw GPS never leaves the controlled archive path, so data science has nothing sensitive to misuse.
- Arrival-time design
- Partitions by when the row landed. A truck's buffered hour shows up as 'now', so the route jumps and training data is wrong in ways nobody notices.
- Event-time design
- Partitions by when the device emitted the ping. A late hour reprocesses only its own partition, and the route comes out smooth.
Common follow-up questions
- A truck replays a day of pings after `location_masking` already ran for that day. What picks them up? (Tests event-time partitions: the late events land in yesterday's partition, and rerunning that partition overwrites the masked output.)
- Data science wants to study dwell time at specific stops. What can they see without raw GPS? (Tests whether the masked output can carry distance-from-stop or cell-level features instead of coordinates.)