The Analyst Who Saw the Salary Data

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain

Pipeline Design

Difficulty

hard

Seniority

L5

Asked in

Epsilon3

Problem

Five business units of very different sensitivity share one flat lake store behind a single reader role, and the platform lead just learned an analyst read another unit's files directly. Redesign the lake so each unit gets its own isolated store, while about 300 analysts and the few pre-approved cross-unit reads keep working.

Two incidents. One shared lake. The access model was never designed, just assumed.

Business requirements

• Two prior incidents already exposed one business unit's data to another, including someone in marketing pulling salary records; the storage layout itself has to keep the units apart, not a single shared lake.
• Marketing legitimately needs the customer dimension and operations needs finance order data; broad direct access for those cases is what failed before, so a governed query engine has to sit between the stores and the consumers and hand back only the approved columns.
• One incident was caused by a pipeline writing into the HR area by mistake; writes have to land only in the unit store they are meant for, so each unit store is fed by its own write path.

Worked solution and explanation

Why this problem exists in real interviews

Five business units with different sensitivity levels share a lake, and there have already been two cross-unit exposures. This is a tenancy-boundary problem dressed up as a storage cleanup. The trap is bolting application-layer filtering onto data that is still commingled. What actually holds is per-unit storage isolation, with each write boundary set at ingest.

The default reach is to keep everything in one store and rely on a single shared reader role plus query-layer filters to keep teams apart. Analysts then read across boundaries by opening files directly, which is how a marketing analyst ends up with a salary file. A single loader with broad write permissions is how one misconfigured pipeline writes into the HR area.

Walk the requirements

Step 1: Give each unit its own store

HR, finance and marketing each land in their own object store, and the storage policy enforces read access there. An analyst who tries to open HR files fails because storage denies the request, not because a query filter happened to catch it. Without per-unit stores, the boundary lives only in the query layer, and both prior incidents went around that layer.

Step 2: Serve cross-unit reads through a column-governed engine

Marketing legitimately needs the customer dimension and operations needs finance order data. Granting either team broad access to the other unit's store is exactly what failed before. Instead, the query engine exposes only the approved columns to the cross-unit role, and the underlying data stays locked. Analysts reach every store through this engine and never directly. Without the engine, the column boundary has nowhere to live.

Step 3: Give each unit store its own write path

Each unit has its own loader, and that loader's credentials can write to one store only. A loader pointed at the wrong store fails at the permission boundary, not at code review. One shared loader with broad write access turns a one-line config error into data landing in another unit's store. Separate loaders keep the blast radius to one unit.

The shape that fits

Common follow-up questions

• Marketing's approved access to the customer dimension needs to grow by one column. What changes, and where? (Tests whether the candidate sees the column grant in the query engine as the place to make this change. The store and its permissions do not change.)
• A new business unit is added. What changes in this design, and what doesn't? (Tests whether the candidate sees a new unit as a new store, a new write-scoped loader, and any new column grants. The engine and the isolation pattern stay as they are.)