The Sale That Needs to Land Now
A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- medium
- Seniority
- L5
- Asked in
- Farfetch
Problem
Design the order pipeline for a global luxury marketplace whose sales arrive in three shapes: web storefront JSON events, mobile apps' batched JSON, and partner integrations polled as XML. Flash sales push order volume to about 20 times the normal 40,000-a-day rate for an hour. A clean daily archive for data science modeling is also required.
Three channels feeding one view. Not all of them speak the same language.
Business requirements
- Merchandising adjusts campaigns and discounts on live performance; sales from every channel have to land in their dashboard within minutes, including during a flash sale, so no channel can sit behind a daily batch.
- Web, mobile, and partner sources each send orders in different formats; downstream consumers can't branch on the channel for every order.
- EU customer email and address are PII under GDPR; raw values can't sit in any queryable table.
Worked solution and explanation
Why this problem exists in real interviews
Multi-channel sales with three properties to fit: minutes-fresh merchandising even during a flash sale, a canonical shape across web, mobile, and partner formats, and GDPR PII isolation. The trap is one ingester per channel that branches downstream and treating PII as a downstream filter.
The default reach is per-channel ingesters and downstream consumers branching on channel. The first flash sale pushes event rates to 20 times normal and the merchandising path can't keep up because the channels run on shared compute. PII lands in the warehouse and is masked in BI views; one direct query exposes raw email. Each channel format change ripples through every consumer.
Walk the requirements
Step 1: Merchandising sees minutes-fresh sales, even in a flash sale
Sales events flow through tokenization and a streaming consumer that updates the merchandising store within minutes. The partner channel is polled, but its output joins the same streaming path, so partner sales reach the dashboard within minutes of being polled, not a day later. A queue between sources and the consumer buffers flash-sale spikes; the consumer catches up with backpressure visible in queue depth rather than dropped events. Without a streaming tier the merchandising freshness is unattainable; without a buffer the spike crashes the consumer.
Step 2: Canonical shape across web, mobile, and partner
Each channel normalizes to a canonical sale shape (channel, customer_token, products, amounts, currency, timestamp) at ingest. Downstream consumers read one schema; a new partner integration adds a normalizer mapping rather than a consumer change. A 'branch on channel everywhere' design is the version where every channel change touches every consumer; canonical-up-front is what keeps consumers stable.
Step 3: Tokenize PII at ingest; raw values never land queryable
EU customer email and address are GDPR PII. Tokenization runs at the boundary; the warehouse and merchandising store hold tokens. The mapping vault sits in a restricted environment with audited access. A 'mask in BI' approach is the version where a direct query reveals the raw values; tokenizing at ingest keeps PII out of queryable tables.
The shape that fits
- web_sales (source, Kafka)
- mobile_sales (source, Kafka)
- partner_sales (source, API)
- tokenizer (transform, Spark)
- tokenization_vault (storage, PostgreSQL)
- canonicalizer (transform, Flink)
- sales_bus (queue, Kafka)
- merchandising_store (storage, PostgreSQL)
- sales_warehouse (storage, warehouse)
- merchandising_team (consumer, Grafana)
- data_science (consumer, Jupyter)
- web_sales -> tokenizer
- mobile_sales -> tokenizer
- partner_sales -> tokenizer
- tokenizer -> tokenization_vault
- tokenizer -> canonicalizer
- canonicalizer -> sales_bus
- sales_bus -> merchandising_store
- sales_bus -> sales_warehouse
- merchandising_store -> merchandising_team
- sales_warehouse -> data_science
- Sources + tokenize: web_sales, mobile_sales, partner_sales, tokenizer, tokenization_vault
- Canonicalize + bus: canonicalizer, sales_bus
- Stores + consumers: merchandising_store, sales_warehouse, merchandising_team, data_science
Common follow-up questions
- A new partner integration sends a format the canonical shape doesn't anticipate. What in this design extends, and what doesn't? (Tests whether the candidate sees the canonicalizer's mapping as the extension point: a new normalizer for the partner, the canonical shape unchanged, and downstream consumers unaffected. Adding fields to the canonical shape (additive evolution) is a separate decision tied to consumer needs.)
- A GDPR deletion request lands. What in this design lets the deletion physically remove the customer from every store? (Tests whether the candidate sees the tokenization vault as the boundary: deletion removes the customer's token mapping, and the warehouse and merchandising store hold tokens that are now orphaned. The data is effectively forgotten because the link to the person is gone, with the orphaned tokens compacted on a schedule.)