Seconds and Months

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

medium

Seniority

mid

Asked in

Autodesk

Problem

Design the pipeline that serves concurrent-seat enforcement, monthly metered-usage billing and daily feature-adoption reports from one telemetry stream. Every desktop and browser session of our design and engineering software feeds that stream, about 2 billion events a day.

Business requirements

• Licensing here. If a customer is over their concurrent-seat limit, we need to know within a few seconds, and presence in a shared document has to feel live.
• Finance here. Metered feature usage goes straight onto the monthly invoice, so a duplicated or dropped event is a billing error a customer will dispute.
• Product analytics here. We look at feature adoption day over day; we do not need it to the second, we need it correct and queryable.
• Platform here. There is one firehose of telemetry; do not stand up a second collection path just for finance.

Worked solution and explanation

Why this problem exists in real interviews

One firehose of telemetry, two consumers with opposite correctness budgets. Licensing wants seat counts within seconds and is happy with approximate as long as it converges; finance wants every metered event counted once and does not care if the number lands a day later. The trap is picking a single clock: stream everything and you pay streaming-scale cost for a billing job that had no latency requirement, or batch everything and licensing enforces seats on yesterday's data. The interviewer is watching whether you justify the path per consumer instead of reaching for one default.

The naive design aggregates all 2 billion daily events in one streaming job that writes a single 'usage' table both consumers read. Licensing is happy. Then a producer retries during a deploy, an offset replays, and the same billable event is summed twice; finance sends an invoice a customer disputes. Meanwhile the streaming bill is several times what it needed to be because the daily adoption report is riding the real-time path for no reason.

Break down the requirements

Step 1: Pin the latency budget per consumer

Licensing is 'within seconds' and tolerates approximate-but-converging counts. Billing is monthly with daily reconciliation and demands exactness. Adoption reporting is daily and demands correctness, not freshness. Write those three budgets down before drawing anything: they decide which path each consumer lives on, and they are the justification the interviewer is grading.

Step 2: Fan out one ingest, do not collect twice

Both paths read the same durable queue. The stream processor consumes it continuously for licensing; the batch path reads the same topic on a daily schedule for finance and product. A second collection path for finance is duplicated operational surface and a place for the two consumers to silently disagree on what happened.

Step 3: Make billing exactly-once at the sink, not in hope

Each event carries a stable client-assigned event_id. The billing job drops every repeat of an event_id before it sums, so a producer retry or a replayed Kafka offset cannot double-count. Anchor the billable period to the event timestamp, not arrival time, so a desktop session that flushes hours late still bills once and into the month it actually happened.

The reference architecture

The split is the whole answer: licensing gets a streaming path into a key-value serving store it queries at seat-check time, and finance plus product get daily Spark jobs into the warehouse. Both read the one Kafka topic, so there is a single source of truth and a single collection cost. The billing dedup is the only place exactly-once machinery lives, because billing is the only consumer that contractually needs it.

Stream everything

One always-on job aggregates all 2B events for every consumer. Simple topology, but you pay real-time compute for daily reporting, and a replayed offset double-bills because there is no dedup boundary.

Path per consumer

Streaming carries only licensing; billing and reporting run daily, with billing deduplicated on event_id. More moving parts, but each consumer is served at its real cadence and billing is exactly-once by construction.

Common follow-up questions

• A new fraud team wants to flag impossible usage patterns (a license key active in five regions at once) within a minute. Which path does it join, and what changes? (Tests whether the candidate adds a stateful streaming consumer off the existing queue rather than rebuilding, and whether they see it as a third latency tier.)
• Event volume 10x's after a free-tier launch and the streaming serving store starts lagging at morning peak. What do you change first? (Tests capacity reasoning: repartitioning the queue, scaling the stream job, and isolating the licensing path so a volume spike does not also stall billing.)
• Finance asks for every billable event's lineage when a customer disputes an invoice. Where do you go? (Tests whether the dedup-by-`event_id` design and event-time anchoring give an auditable, replayable record rather than an irreproducible streaming snapshot.)