Half a Million Rental Cars

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

Our 500,000 rental vehicles stream telematics continuously while a reservation system records rentals and service centers log maintenance, and no view spans all three. Design the pipeline and warehouse that unify the three sources on vehicle id and serve each consumer at its own freshness.

Every vehicle is reporting. Every rental matters.

Business requirements

• Operations dispatches off live vehicle position and needs minutes, rental visibility tolerates up to half an hour, and maintenance scheduling tolerates a day.
• Vehicles change depot, accumulate damage, and rack up odometer readings; a historical rental has to report the vehicle's depot, damage status, and odometer as they were on that date.
• When a vehicle leaves where it's supposed to be, enters a restricted zone during a rental, or sits parked for too long, ops needs to act quickly.

Worked solution and explanation

Three clocks and one time machine

This is three freshness budgets and a point-in-time join dressed up as fleet visibility. Anyone can draw a Kafka box and a warehouse. The trap is that rentals sit in the middle: not live like dispatch, not nightly like maintenance. The word 'batch' is how candidates bury them. Rental visibility has a 30 minute budget, so a rentals path on an hourly or nightly cadence fails the requirement even though it looks like the sensible slow lane.

Step 1: Give each consumer its own clock

Telematics fans out to ops_stream, which keeps ops_store within a minute for dispatch, and to zone_detector, which watches for breaches. Reservations take a separate quarter-hour path. Maintenance and vehicle history run daily. One shared cadence under-serves at least two of the three consumers.

Step 2: Keep rentals inside 30 minutes

reservations arrives by CDC and rentals_sync processes it continuously, tagged '< 15min', straight into unified_warehouse. The warehouse carries no daily SLA, because a '< 24h' tag on the node rentals pass through quietly turns the whole rentals path into a next-day path.

Step 3: Version the vehicle, then join as of the rental date

vehicle_history writes a new row to versioned_vehicle_dimension whenever depot, damage or odometer changes, keyed on vehicle_id with valid_from and valid_to. rentals_sync reads that dimension and picks the row whose window contains the rental date. That is how a rental last March still reports March's depot.

Step 4: Detect on the stream and page someone

zone_detector reads telematics plus reservations, so it knows which vehicles are mid-rental, and it keeps per-vehicle idle state. Breaches go straight to ops_alerts, an alerting consumer that pages on-call. A nightly scan would tell ops about a restricted-zone entry the next morning.

Join to today's vehicle

A plain equi-join on vehicle_id is cheap, but every depot change silently rewrites every past rental. Last quarter's utilization by depot changes when you rerun it.

Join as of the rental date

Match rental_date between valid_from and valid_to. The join costs more and the dimension grows, but a historical report is reproducible.

Common follow-up questions

• A vehicle changes depot mid-rental. What does the rental report, and what does ops see? (The rental fact uses the row valid on its date. The live `ops_store` shows the current depot. Both are correct for different questions.)
• An idle alert fires while the vehicle is in the service bay. What would you change? (Today `zone_detector` reads only telematics and reservations; feeding it `maintenance_events` lets an active service visit suppress the parked alert.)
• A rental arrives 40 minutes late from CDC, then is resent after an outage. What happens to the fact? (Tests whether `rentals_sync` writes idempotently and whether late rows reuse the same as-of dimension lookup.)