# The Dashboard and the Attribution Model

> Streaming and batch. One pipeline to rule them.

Canonical URL: <https://datadriven.io/problems/the_dashboard_and_the_attribution_model>

Domain: Pipeline Design · Difficulty: hard · Seniority: L6

## Problem

Design one architecture on Azure Databricks that feeds both a real-time campaign performance dashboard and a daily attribution report from a stream of ad impression and conversion events peaking near 200,000 per second. Today separate streaming and batch pipelines produce the two, and their numbers have drifted apart.

## Business requirements

- The daily attribution report is what we bill against and the live dashboard has to show the same numbers for the same period; today they diverge.
- Mobile impressions arrive hours late; both the dashboard and the daily report have to attribute each event to the time it actually happened.
- Attribution credits the last impression seen within the lookback window before a conversion; the daily report has to read the relevant historical window for each day's conversions.
- Today the dashboard's approximate dedup misses duplicates that arrive late while the batch report removes them exactly; the unified view can't keep that gap.

## Worked solution and explanation

### The trap

This is a consistency problem wearing a streaming-vs-batch costume. Anyone can draw a fast path for the dashboard and a slow path for the report. That is exactly the architecture that is drifting today. The real skill is making the daily report a **closed-window read of the same rollups** the dashboard reads, so the two views have nothing to disagree about. Keep two computations and every rule gets written twice: dedup, lateness, the attribution lookback. The invoice you bill against then stops matching the number the client watched all day.

### Build it in four decisions

#### Step 1: Dedup once, upstream of both views

Drop duplicates on a stable event id in `dedup_stream`, before anything aggregates. Today the streaming side dedups approximately and the batch side dedups exactly, and that gap is where the dashboard overcounts. A duplicate that arrives late is still caught, because dedup keys on the id and not on arrival order.

#### Step 2: Window on event time, with a lateness allowance

`event_time_attribution` buckets by when the impression happened, not when it arrived. A mobile event that shows up three hours late updates its own hour through the `upsert` into the warehouse, and because both views read that bucket, both restate together.

#### Step 3: Give attribution the history it needs

Last-touch credit needs the impressions from before today. `event_archive` holds the deduped history, and `event_time_attribution` reads the lookback window from it for each conversion. A same-day-only lookup silently drops credit for yesterday's impression.

#### Step 4: Serve both views from one warehouse table

Rollups land in `rollup_warehouse`. The dashboard reads the open periods and the report reads yesterday's closed window. Same rows, two queries, no reconciliation job.

### The reference design

**Pipeline stages**

- impression_events (source, Kafka)
- conversion_events (source, Kafka)
- dedup_stream (transform, Flink)
- event_archive (storage, object storage)
- event_time_attribution (transform, Flink); idempotency: upsert
- rollup_warehouse (storage, Databricks)
- dashboard (consumer, Grafana)
- daily_report (consumer, reporting)

**Data flow**

- impression_events -> dedup_stream
- conversion_events -> dedup_stream
- dedup_stream -> event_archive
- dedup_stream -> event_time_attribution
- event_archive -> event_time_attribution
- event_time_attribution -> rollup_warehouse
- rollup_warehouse -> dashboard
- rollup_warehouse -> daily_report

*One dedup, one event-time attribution job, one `rollup_warehouse`: the dashboard and `daily_report` are two reads of the same rows.*

**Two synchronized pipelines**

Stream dedups approximately, batch exactly. Stream buckets by arrival, batch by event time. Each owns its own attribution window. Numbers diverge, and someone writes a reconciliation job.

**One computation, two reads**

`dedup_stream` and `event_time_attribution` run once. The report is a filter on `rollup_warehouse`, so any disagreement would have to come from a query, not a pipeline.

> **Sharing code is not sharing a computation**
>
> Candidates often answer with a shared attribution library called from both a streaming job and a batch job. That still gives two runs over two inputs with two dedup states, so the views still drift. The fix is one output table, not one codebase.

> **Say where late data lands**
>
> The senior tell is naming the lateness allowance and what happens past it. An event inside the allowance rewrites its bucket through the `upsert` from `event_time_attribution`, and both views see it on their next read. An event past it still sits in `event_archive`, so that day can be recomputed once and both views restate together.

## Common follow-up questions

- An impression arrives a week late, past the lateness allowance. What do the dashboard and the report show? _(Tests a replay path from `event_archive` that recomputes the affected day once, so both views restate the same period.)_
- The attribution lookback grows from 30 to 90 days. What gets expensive? _(Tests whether the candidate sees that attribution state and archive reads scale with the window, and how partitioning by event date keeps the read bounded.)_

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/the_dashboard_and_the_attribution_model)
- [System Design Interview Questions](https://datadriven.io/data-engineering-system-design)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.