One Earthquake, Ten Thousand Tweets

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L5

Problem

We detect breaking events like earthquakes and fires from the full Twitter firehose and 1 million other data sources, where one event surfaces as thousands of simultaneous posts. Within 60 seconds, we send a single validated alert to our clients. Design the event detection and deduplication pipeline, which today misses multi-source signals.

The firehose is on. Separate signal from noise.

Business requirements

• Hedge fund and government clients pay for the fastest alerts; a slower delivery costs them their competitive advantage.
• An earthquake produces ten thousand posts about the same event; clients receive one alert per event, not ten thousand.
• Government clients act on alerts and a false positive can drive a real-world emergency response; the published false-positive rate has to stay very low.

Worked solution and explanation

This is an event-identity problem dressed up as a low-latency alert feed. Anyone can put Kafka in front of a stream job. The skill being probed is whether you treat speed and confidence as two separate dials, and whether you key everything on the event instead of the post. Get that wrong and an earthquake pages a hedge fund ten thousand times, or a phantom fire sends a government agency into an emergency response.

Walk the requirements

Step 1: Cluster on the stream, before anything is stored

A 60-second budget leaves no room for a batch hop, so event_dedup_clusterer is a Flink job with real-time freshness. It groups posts by location, time window and content similarity into one event candidate. Text similarity alone fails: ten phrasings of the same quake look like ten events. Corroboration accumulates on the cluster, so the signal count rises while the event count stays at one.

Step 2: Publish per tier, from the event and not the post

tier_publisher reads event candidates from event_store and applies a threshold for each tier. Priority clients get the alert at a lower corroboration bar, within seconds. Government clients wait for the higher bar. Speed and confidence become two inputs to the publish decision, so the false-positive rate for government stays low without slowing the fast tier.

Step 3: Record every delivery against the event id

alert_delivery_log stores one row per event and tier. The publisher checks it before sending, so a restarted job or a replayed cluster update cannot send a second alert for the same event. This log is what makes the one-alert-per-event claim checkable.

One shared threshold

Fast enough for priority means too eager for government. A false positive at the speed threshold reaches an agency that acts on it. Raise the bar to protect government and you slow every paying client.

Per-tier thresholds

tier_publisher fires priority at low corroboration and government at high corroboration. The same event can reach priority in seconds and government a little later. Neither tier pays for the other's requirement.

Common follow-up questions

• Two real events land close together in place and time, such as a fire next to a power outage. How does the clusterer avoid merging them? (Tests multi-feature clustering: location and time alone over-merge, so content similarity has to disagree before clusters join.)
• A late batch of posts arrives after the priority alert has already gone out. What happens? (Tests late data: the cluster updates under the same event id, and only a tier that has not yet been sent can fire.)