The User Who Asked to Be Forgotten

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L7

Problem

Design the end-to-end pipeline that turns listener events from mobile, web and smart-speaker clients into dashboards showing hourly and daily engagement metrics. The events arrive as one continuous stream of about 50,000 per second.

Users want their data erased. Completely.

Business requirements

• GDPR requires that a user's data be deleted within 30 days of their request, across every system that holds it.
• Mobile clients buffer events for tens of minutes when connectivity is bad; engagement metrics have to count those events for the time the user listened, not when the platform received them.
• Client retries deliver the same event twice; engagement and royalty numbers can't be inflated by those retries.
• Major live events drive sudden 15x volume spikes that have caused the pipeline to fall behind in the past.

Worked solution and explanation

What this really is

This is a deletion control plane dressed up as an engagement dashboard. Anyone can draw events into a stream processor into a warehouse. The real test is whether erasure is a path with receipts or an afterthought. An hourly aggregate carries no user id, so you cannot erase one listener from a count. You have to scrub the raw archive and recompute every bucket that counted them. Skip that and day 31 arrives with a clean archive, contaminated aggregates, and nothing to show the regulator.

Walk the requirements

Step 1: Buffer before you compute

A 15x live-event spike belongs in event buffer as consumer lag, not in the aggregator as dropped events. Kafka holds the burst durably while Flink, sized for sustained load, catches up within minutes and nothing is lost.

Step 2: Dedup on the client event id before anything is counted

Retries resend the same identifier. event dedup drops the second copy with keyed state, upstream of both the aggregator and the archive, so live counts and replays see the same events. Royalties come from these numbers, so a few percent of inflation is real money.

Step 3: Window on event time with an allowed lateness

Phones buffer events for tens of minutes. windowed aggregator buckets on the client timestamp and holds windows open for the lateness allowance. Anything later flows through the archive and the compactor into its true hour. It must never land in the hour it happened to arrive.

Step 4: Give each store its own deletion receipt

deletion orchestrator fans each request out to the archive and to the aggregates. It records a confirmation per store and alerts when one is overdue. The 30-day proof is that set of receipts.

Arrival-time buckets

A phone reconnects at 9:40 and drains 30 minutes of listening. The 9:00 hour spikes, 8:00 comes up short, and the dashboard reports behavior that never happened.

Event-time buckets

The same drain lands in 8:00, where it belongs. Anything past the allowance takes the event archive to late data compactor path and overwrites the 8:00 partition.

Common follow-up questions

• One store is offline when a deletion request arrives. What does `deletion orchestrator` report? (The request stays open with per-store status, retries run, and an alert fires before the 30-day deadline. The answer is never a silent pass.)
• How do you erase a user whose events sit in compacted Parquet files in `event archive`? (Tests partitioning by event hour so the rewrite touches only affected files, followed by a targeted overwrite of those hours in `engagement metrics`.)