A Million Moving Dots
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- mid
- Asked in
Problem
We run a delivery marketplace where every active courier streams a location ping every few seconds, tens of billions of events a day at peak. Customers need a live map and ETA that updates within seconds of each ping, while finance settles courier payouts and merchant fees once a day and cannot tolerate a double-counted or dropped delivery. Design the platform that serves both the live tracking and the exactly-once daily settlement off the same firehose.
Business requirements
- As a customer I want to watch my courier move on the map and see an ETA that updates within seconds of where they actually are.
- As finance I need each completed delivery counted exactly once when we pay couriers and bill merchants; a duplicate or a dropped delivery is a real money error.
- As an on-call engineer I need the platform to survive dinner-rush spikes without losing events when a downstream consumer slows down.
- As a platform owner I cannot let one malformed or out-of-order ping from a buggy app version stall everyone else's tracking.
Worked solution and explanation
Why this problem exists in real interviews
Behind the friendly phrase 'support a high number of deliveries' are two consumers reading the same firehose with opposite correctness budgets. The customer map wants the courier's position within seconds and is happy with approximate. Finance wants every completed delivery counted exactly once at end of day and a duplicate is real money. The trap is one pipeline that tries to serve both: aggregate hard enough to settle accurately and the map lags; optimize for the live map and settlement double-pays on the first consumer retry.
The other half of the trap is the firehose itself. Tens of billions of pings a day arrive in dinner-rush bursts. The whiteboard answer of 'app writes straight to a database' falls over the moment a downstream consumer slows down, and a single malformed ping from a bad app version stalls everyone if there is nowhere for poison events to go.
Break down the requirements
Step 1: Buffer the firehose before anything reads it
Producers (millions of phones) and consumers move at different speeds. A partitioned, durable message queue decouples them: bursts are absorbed, offsets are retained so a fixed consumer can replay, and no single slow sink drops events. Partition on a high-cardinality key (delivery or courier id) so one popular city does not create a hot partition.
Step 2: Stream the live map; approximate is acceptable
A stream processor consumes positions and upserts the latest into a low-latency serving store the customer app queries at high QPS. The SLA that matters is a few seconds end to end. Out-of-order pings resolve with latest-known-position wins; you are not reconstructing an exact path, you are answering 'where is my courier right now'.
Step 3: Settle exactly once, off the completion event
Do not settle off the raw pings; settle off the discrete delivery-completion event, which carries a stable delivery_id. Dedup on that id and write idempotently into the warehouse (upsert or partition-overwrite). Now a consumer retry or a replay re-produces the same row instead of paying a courier twice. This is the requirement finance and on-call care about from opposite directions: drop one and a courier is underpaid, double-count one and the payout is wrong.
Step 4: Give poison events somewhere to go
Validation lives on the stream processor. A ping that fails schema or parsing routes to a dead-letter queue and the main consumer acks and keeps draining. Without it, one buggy app version's malformed events stall the live map for every customer until someone patches the parser.
The shape that fits
- location_pings (source, Kafka, 30B events/day, high-cardinality partitioning)
- tracking_stream (transform, Flink, real-time, errors to DLQ)
- dlq (storage, S3, alert on depth above baseline)
- live_position_store (storage, Redis, < 1min, latest-position upsert)
- customer_map (consumer, Mobile app, < 1min)
- completion_events (queue, Kafka)
- settlement_batch (transform, Spark, dedup on delivery_id, staging table)
- settlement_warehouse (storage, BigQuery, < 24h, idempotent upsert)
- finance_dashboard (consumer, Looker, < 24h)
- location_pings -> tracking_stream
- tracking_stream -> dlq
- tracking_stream -> live_position_store
- live_position_store -> customer_map
- location_pings -> completion_events
- completion_events -> settlement_batch
- settlement_batch -> settlement_warehouse
- settlement_warehouse -> finance_dashboard
- Ingest: location_pings
- Live path: tracking_stream, dlq, live_position_store, customer_map
- Settlement path: completion_events, settlement_batch, settlement_warehouse, finance_dashboard
- One pipeline for both
- Aggregate the firehose into one metrics table the map and finance both read. The map lags because it waits on the heavier settlement aggregation, and settlement double-counts on any consumer retry because nothing deduped on a stable id.
- Split by budget
- Stream the map off latest-position-wins for a few-second SLA; settle from the discrete completion event, deduped and idempotent, on a daily batch. Each consumer gets exactly the correctness and latency it needs.
Common follow-up questions
- Dinner rush in one metro spikes to 25x normal volume for that city. How does the platform absorb it without dropping pings or stalling other cities? (Tests partitioning strategy, consumer auto-scaling, and whether the durable buffer isolates a hot region from the rest.)
- A delivery's completion event arrives twice with the same `delivery_id` but a different final amount. What protects you and what does not? (Tests whether the candidate sees that dedup on id absorbs identical duplicates but a mutating duplicate needs a conflict policy, not just a unique key.)