The Fare Aggregator
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- L5
Problem
Design the pipeline behind flight search and booking for 100 million users, fed by 80 billion fare checks a day from dozens of airline APIs and global distribution systems. The pricing team sees fares go stale within seconds in booking surges, and no two sources agree on fields: departure_time here, dep_utc there, a Unix timestamp elsewhere.
Airfares shift every minute. Catch the best ones.
Business requirements
- Users on flight search expect a result within a couple seconds, and the price they see at booking has to be quoted live from the source at that moment, not read back from the search cache.
- Around fifty GDS and airline sources each use their own fields and tax breakdowns; downstream search and booking can't branch on the source.
- When a flight is delayed or cancelled, users actively on the booking page for that route have to be alerted before they pay.
Worked solution and explanation
Why this problem exists in real interviews
Aggregating airfares from many GDSs and APIs into one search experience under a couple-second latency, with disruption alerts to users actively on the booking page. The trap is per-source branching downstream of search and treating disruption as a separate batch system that finds out about cancellations after users do.
The default reach is for search and booking to branch on source schema. Each source change ripples through search and booking; the first format change costs a quarter to land. Disruption notifications run in a separate nightly system; users who are actively on the booking page when a flight cancels find out at the airport.
Walk the requirements
Step 1: Canonical fare shape on the bus; downstream reads one schema
Each source's events normalize to a canonical (origin, destination, departure_time, fare, taxes, source) shape at ingest; the bus carries canonical events. Search and booking read the canonical shape; a new source adds a normalizer mapping, not a search-or-booking change. A 'branch on source' design is the version where every source change ripples through every consumer; canonical-up-front is what keeps the consumers stable.
Step 2: Fast search from the store, live price at booking
Search queries hit a low-latency store fed by the streaming canonicalizer, so results come back inside the page's budget without a request-time fan-out to fifty sources. That store is still a copy, so booking does not trust it alone: a live recheck takes the held itinerary from the store and the newest canonical quote straight from the canonicalizer, which reads the sources live, and the booking page shows that price. Booking off the store alone is the version where a surge moves the fare between search and pay.
Step 3: Disruption events route to active booking sessions
When a flight is delayed or cancelled, the airline emits a disruption event onto the bus. A session-aware consumer matches it to users currently on the booking page for the affected route and pushes an alert before they pay. A 'nightly disruption batch' is the version where users find out at the airport; the streaming path matched to active sessions is what makes the alert actionable.
The shape that fits
- gds_apis (source, API)
- airline_apis (source, API)
- canonicalizer (transform, Flink); freshness: real-time
- fare_bus (queue, Kafka)
- search_store (storage, PostgreSQL); freshness: real-time
- live_price_recheck (transform)
- disruption_consumer (transform, Flink); freshness: real-time
- active_sessions (storage, PostgreSQL)
- search_consumer (consumer, API)
- booking_page (consumer, API)
- booking_alerts (consumer, API)
- gds_apis -> canonicalizer
- airline_apis -> canonicalizer
- canonicalizer -> fare_bus
- fare_bus -> search_store
- fare_bus -> disruption_consumer
- active_sessions -> disruption_consumer
- search_store -> search_consumer
- search_store -> live_price_recheck
- canonicalizer -> live_price_recheck
- live_price_recheck -> booking_page
- disruption_consumer -> booking_alerts
- Sources + canonicalize: gds_apis, airline_apis, canonicalizer, fare_bus
- Search, booking + disruption: search_store, live_price_recheck, disruption_consumer, active_sessions
- Consumers: search_consumer, booking_page, booking_alerts
Common follow-up questions
- A new GDS source signs on with a more complex tax breakdown than the canonical shape. What in this design lets it land without changing search or booking? (Tests whether the candidate sees the canonical shape's tax breakdown either accommodating the new source's structure (additive evolution) or the canonicalizer mapping the new source down to the canonical shape with the extra detail dropped or summarized. Search and booking don't change; the canonicalizer absorbs the difference.)
- Disruption events for a route burst as a major weather event hits. What does this design do, and how does it avoid notifying users twice? (Tests whether the candidate sees the disruption consumer dedup on (flight, `event_type`) and notify each session once per route per event. Active-session state tracks notified status. Without dedup, users would get repeated notifications and tune them out.)