Recommendations Now, Royalties Later

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

How would we build one pipeline that takes the viewing events from our video service, around 200,000 per second at evening peak across thousands of licensed and original titles, and feeds both the recommendation system and the licence reporting that studios are paid from?

The catalog updated. Did anyone notice?

Business requirements

• What the customer just watched changes their next recommendation; the recommendation system needs that signal within minutes, while licence reporting tolerates T+1.
• Studios are paid by exact viewing minutes per title per country; licence reports can't double-count or miss minutes.
• Title metadata changes when licences expire and titles are added or removed; reports have to use the metadata that was current when the user watched, not today's.

Worked solution and explanation

Why this problem exists in real interviews

This is two correctness budgets dressed up as one event stream. Recommendations want what you just watched within a minute and can live with approximate. Licence reporting can wait until tomorrow but has to be exact, and it has to price each view with the title metadata that was current when it happened. Everyone draws the streaming path. What separates candidates is refusing to let the payout numbers come out of it.

The default reach is one streaming aggregator whose counts table both consumers read. Recommendations are fine. Studios get paid from approximate counts that miss late offline syncs and double-count retried events. The reports also join today's catalog, so a licence that expired last week still looks valid for last week's views.

Walk the requirements

Step 1: Size each path to its reader

Events flow from the topic straight into a stream processor that updates per-viewer features in a serving store the recommender reads at request time. The same events land in object storage partitioned by event time. If recommendations had to wait for the batch, what you just watched would not be a signal yet. If licence counts came from the stream's state, studios would be paid the wrong number.

Step 2: Make licence minutes exact at the boundary

The daily batch reads the archive over the late-arrival window, drops duplicates on the event id, and counts minutes per title per country. Offline devices sync up to 48 hours late, so the batch recomputes those days and overwrites their partitions instead of appending. A late minute lands in the day it was watched, and a rerun never doubles it.

Step 3: Join metadata as of the viewing time

Title metadata is kept as history with title_id, valid_from and valid_to. The enrichment stage joins each deduplicated view on viewing_time BETWEEN valid_from AND valid_to. A title licensed during that window gets paid for its views. One that expired before the view does not, even if it is back in today's catalog.

The shape that fits

Common follow-up questions

• An event arrives 50 hours late, past the late-arrival window. What does this design do, and what does the licence report show? (Tests whether the candidate makes the window boundary explicit. The event is either dropped or routed to an exception path the team triages, and studios are paid against the period as it stood when the window closed.)
• A title's licence terms are restated retroactively. What changes for licence reporting, and what doesn't? (Tests whether the candidate adds a corrected metadata version with a new `valid_from` and reruns the affected days. The recommendation path is untouched.)