A Million Moving Dots

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain: Pipeline Design
Difficulty: medium
Seniority: mid
Asked in: Google

Problem

Every active courier's phone emits a location ping every few seconds, tens of billions a day at peak, while only a few million deliveries complete daily. That one stream drives the customer's live map and ETA and the once-a-day settlement of courier payouts and merchant fees. Design the platform behind both.

Business requirements

As a customer I want to watch my courier move on the map and see an ETA that updates within seconds of where they actually are.
As finance I need each completed delivery counted exactly once when we pay couriers and bill merchants; a duplicate or a dropped delivery is a real money error.
As an on-call engineer I need the platform to survive dinner-rush spikes without losing events when a downstream consumer slows down.
As a platform owner I cannot let one malformed or out-of-order ping from a buggy app version stall everyone else's tracking.

Worked solution and explanation

Why this problem exists in real interviews

Underneath the delivery story are two consumers reading one firehose with opposite correctness budgets. The customer map wants the courier's position within seconds and is happy with approximate. Finance wants every completed delivery counted exactly once at end of day, and a duplicate is real money. The trap is one pipeline that tries to serve both: aggregate hard enough to settle accurately and the map lags; optimize for the live map and settlement double-pays on the first producer retry.

The other half of the trap is the firehose itself. Tens of billions of pings a day arrive in dinner-rush bursts. The whiteboard answer of 'app writes straight to a database' falls over the moment a downstream consumer slows down. A single malformed ping from a bad app version also stalls everyone if there is nowhere for poison events to go.

Break down the requirements

Step 1: Buffer the firehose before anything reads it

Producers (millions of phones) and consumers move at different speeds. A partitioned, durable message queue decouples them: bursts are absorbed, offsets are retained so a fixed consumer can replay, and no single slow sink drops events. Partition on a high-cardinality key (courier or delivery id) so one busy city does not create a hot partition.

Step 2: Stream the live map; approximate is acceptable

A stream processor consumes positions and upserts the latest into a low-latency serving store that the customer app queries at high QPS. The SLA that matters is a few seconds end to end. Out-of-order pings resolve with latest-known-position wins: you are not reconstructing an exact path, you are answering 'where is my courier right now'.

Step 3: Settle exactly once, keyed on `delivery_id`

Do not settle off the raw pings. Settle off the delivery-completion event, which carries a stable delivery_id, and upsert into the warehouse on that key. A producer retry, a consumer replay or a rerun of a corrected day re-produces the same row instead of paying a courier twice. Because the buffer retains every event, nothing is dropped either: a failed day is simply re-read.

Step 4: Give poison events somewhere to go

Validation lives on the stream processor. A ping that fails schema checks or parsing routes to a dead-letter queue, and the main consumer acks and keeps draining. On-call can inspect the quarantined events and replay them once the app is fixed. Without this path, one buggy app version stalls the live map for every customer.

The shape that fits

One buffered firehose splits into a real-time streaming path for the live map, with bad pings quarantined, and a daily batch path that upserts each delivery once into the warehouse.

courier_apps (source, API)
event_buffer (queue, Kafka)
tracking_stream (transform, Flink)
dead_letter_queue (queue, Kafka)
live_position_store (storage, Redis); freshness: real-time
customer_app (consumer, API)
settlement_job (transform, Spark)
settlement_warehouse (storage, warehouse); freshness: < 24h; idempotency: upsert

courier_apps -> event_buffer
event_buffer -> tracking_stream
tracking_stream -> live_position_store
tracking_stream -> dead_letter_queue
live_position_store -> customer_app
event_buffer -> settlement_job
settlement_job -> settlement_warehouse

One pipeline for both: Aggregate the firehose into one metrics table that the map and finance both read. The map lags because it waits on the heavier settlement aggregation, and settlement double-counts on any retry because nothing is keyed on a stable id.
Split by budget: Stream the map off latest-position-wins for a few-second SLA. Settle from the completion event, upserted on delivery_id, on a daily batch. Each consumer gets exactly the correctness and latency it needs.

Common follow-up questions

Dinner rush in one metro spikes to 25x normal volume for that city. How does the platform absorb it without dropping pings or stalling other cities? (Tests partitioning strategy, consumer auto-scaling, and whether the durable buffer isolates a hot region from the rest.)
A delivery's completion event arrives twice with the same `delivery_id` but a different final amount. What protects you and what does not? (Tests whether the candidate sees that a key absorbs identical duplicates, but a mutating duplicate needs a conflict policy, not just a unique key.)