The Speed Layer

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

Activity events arrive from web and mobile clients, about 60 million a day from roughly 8 million users, and hourly, daily and weekly active users are currently counted from them at query time, which is too slow. Design a pipeline that pre-computes all three, refreshed every hour, and serves them.

Dashboards can't wait for raw logs. Something has to happen upstream.

Business requirements

• Product wants HAU within minutes of the hour closing and DAU shortly after midnight; today they wait until 2am for yesterday's number.
• DAU goes into investor reporting; the count has to be exact, not approximate.
• Mobile clients buffer events for hours; a late event from yesterday has to update yesterday's numbers, not be silently dropped.

Worked solution and explanation

What this really is

This is a freshness problem and a correctness problem wearing the same metric name. HAU has to land minutes after the hour, DAU goes to investors and must be exact, and mobile clients deliver yesterday's events today. Anyone can draw a stream processor feeding a warehouse. The trap is reaching for HyperLogLog to hit the speed budget, which breaks the investor-exact constraint, or leaning on one nightly recompute, which misses the hourly window and still leaves history wrong.

The default reach is one streaming job keeping approximate sketches, plus a nightly recompute to patch DAU. The sketch number and the recount never agree, so finance cannot file either one. Late mobile events arrive after the daily run, so they are dropped or credited to the wrong day. HAU ends up fresh, DAU is exact only once a night, and history never gets corrected.

Walk the requirements

Step 1: Emit exact counts at every window close

A stream processor holds the unique users per (period_type, period_id) for the hour, the day and the week. When an hour closes, it writes that hour's count to the warehouse; the day closes at midnight UTC the same way. Dashboards read pre-aggregated rows, so nothing is recomputed at query time. Counting raw events at read time is exactly what made product wait until 2am.

Step 2: Hold real user sets, not sketches

Investor DAU cannot be an estimate. The stream processor keeps the real user ids for each open period rather than a probabilistic sketch. Memory scales with the active users in the open windows, which is bounded. A sketch hits the speed budget, but it produces a number that never matches a full recount, and that recount is the figure finance is held to.

Step 3: Give late events a path back into history

Every event carries its event time. The stream processor closes each window on time so HAU stays fresh, while the raw events also land in object storage. A scheduled Spark job recounts any past day or week that a late event touched and overwrites that period's partition, so the corrected figure replaces the original. Dropping events past a lateness cutoff is the version where history goes silently wrong.

The shape that fits

Common follow-up questions

• Active users for a single hour spike past the aggregator's memory budget for an exact set. What does this design do, and what is the failure mode if it does nothing? (Tests whether the candidate sees the memory bound: spill the set to disk (slower but still exact), spread it across more workers, or, only if the business accepts approximation for that grain, use a sketch. The trade should be named, never made silently.)
• DAU for a day a month ago is corrected upward after a late buffer drain, but investor reporting has already filed on the original number. What does this design do? (Tests whether the candidate keeps the as-filed value alongside the restated one, or in an immutable filing archive outside the live warehouse, so the regulatory record survives the correction.)