Eight Teams, Eight Latencies

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

The streaming bill tripled after every player event moved onto real-time jobs, and the platform lead points out that most of the eight teams reading them only look at next-day numbers. Design the pipeline for about 3 billion events a day from 200 million players: matchmaking, trophy unlocks, in-game purchases and session telemetry.

Millions of gamers. The architecture decision changes everything.

Business requirements

• Suspected cheaters must be banned before the match they are currently playing ends (matches last 8-12 minutes).
• In-game purchase revenue must reconcile to the payment processor to the cent, with no duplicated or dropped transactions.
• The eight downstream consumers have freshness needs ranging from 30 seconds to T+1, and pipeline cost is reviewed quarterly against consumer benefit.

Worked solution and explanation

What this problem really is

This is a cost-allocation problem dressed up as a gaming pipeline. Eight teams read the same event firehose, but only a few change the outcome when their data is seconds old, and anti-cheat is the one that cannot wait: a cheater flagged after the match ends has already ruined it. Everyone draws the Kafka topic. What separates candidates is refusing to put every consumer on the same tier. Stream everything and finance and analytics pay streaming prices for numbers they read tomorrow. Batch everything and anti-cheat never bans anyone mid-match.

Building it

Step 1: Land every event once on a durable log

All player events go to one Kafka topic. Its retention is what lets both tiers read the same facts, and what lets a failed batch load replay instead of losing data.

Step 2: Give anti-cheat its own real-time path

anti_cheat_stream is a stateful Flink job over sliding windows of each player's actions, set to 'real-time'. It hands enforcement_service a flagged player ID and a reason, so the ban lands within seconds, well inside an 8 to 12 minute match.

Step 3: Put everything next-day on one batch path

batch_dedup_load is a nightly Spark job that loads the day's events into event_warehouse. Finance reconciliation and the analytics dashboard both read there, so the five teams that only need next-day numbers share one cheap path.

Step 4: Deduplicate purchases on their unique ID

Client retries and processor callbacks redeliver purchases, so the load keeps one row per purchase ID before writing the warehouse. Finance then matches those totals against the processor's daily settlement file to the cent; a duplicate inflates revenue and a drop under-reports it.

The reference design

Stream everything

Every consumer gets seconds-old data, and the bill pays for it eight times over. Finance still reconciles once a day against a settlement file, so the freshness buys nothing.

Two tiers

Only anti-cheat runs continuously. Next-day consumers share one nightly load, and purchases are deduplicated once, on the path finance actually reads.

Common follow-up questions

• A tournament brings 25x the usual peak for one title in 48 hours. What changes? (Tests partition headroom, scaling the real-time job, and isolating it from other titles.)
• Analysts change the definition of a daily active user and want it recomputed for the last six months. How? (Tests replay from the durable log or the warehouse, and rerunning nightly loads without double counting.)
• 500+ titles each add their own event fields. How does the schema hold up? (Tests a shared core schema with per-title extensions and compatibility checks at ingest.)