Data Mesh

Dehghani's 2019 article traced the trouble with warehouses and lakes to 3 failure modes, none of them about storage. The platform is centralized and monolithic: 1 system handles every domain's data from ingestion to serving. Its pipelines are split by technical stage (ingest, cleanse, aggregate, serve) rather than by domain, so a new field touches every stage. And its owners are hyper-specialized data engineers who sit apart from both the teams that create the data and the teams that use it.

The result shows up as a queue. Say 12 domains file 50 requests a month to the central team (a new source, a changed column, a broken metric) and the team closes 40. The backlog grows by 10 a month, so after 6 months 60 requests are waiting and a new one sits for about 7 weeks before anyone starts it. Hiring helps only until the next domain arrives, because demand grows with the number of domains and the team's context does not.

Long waits push domains to build their own pipelines outside any standard, and the engineers who do get to a request often model the data wrong because they do not know the domain. Data mesh answers both at once: the people with the context own the data, and a shared platform and shared rules keep their work compatible.

If every domain had to provision its own storage and orchestration, then configure access control and build monitoring on top, the cost of ownership would sink the model. The self-serve platform removes that cost. A platform team builds it as a product for internal users, and its measure of success is how quickly a domain engineer can ship a new data product that meets the global standards.

The 2020 article splits the platform into 3 planes. The data infrastructure provisioning plane handles the low-level resources: storage and compute, plus identities and access. The data product developer experience plane gives domains a simple way to declare a data product and to build and test it before deploying, so they work with data products rather than with buckets and clusters. The mesh supervision plane works across all products: discovery, the knowledge graph of how products relate, and checks that every product follows the global policies.

In practice the first version is usually thin: a template repository per data product, CI that runs tests and contract checks, a shared table format in 1 catalog, a data catalog that products register in on deploy, and default monitoring. That is enough if a domain engineer can publish a new product by writing its transformation and a short declaration, without filing a ticket.

Decentralized ownership without shared rules produces data that can't be joined. Federated governance is a group made of domain data product owners and platform owners that decides which few things must be the same everywhere and leaves everything else to the domains.

What gets standardized is what crosses domain boundaries. The 2020 article calls these polysemes: concepts such as a customer or a listener that several domains refer to, each in its own way. The federation agrees how such an entity is identified, so customer_id in orders and in marketing means the same person. It also classifies personal data and sets the access policy. Table and event formats come from the federation too, along with the metadata every product must publish. How a domain models its own data stays local.

Computational means the rules run as code on the platform instead of living in a document. A deploy that publishes an unclassified personal data column fails in CI. A schema registry refuses an incompatible change to a published version. A product that misses its freshness objective shows up on the supervision plane without anyone auditing it.

Data contracts are the common way to write those rules down per product: a versioned file that states the schema and semantics along with the quality checks and service levels its owner commits to. CI checks every change against it. The Open Data Contract Standard, a Linux Foundation AI & Data incubation project under Bitol, gives that file a shared format, so tools can read the same contract.

Take the orders domain in the reference design. Its product orders.order_placed has 3 consumers, marketing attribution among them. The orders team needs to split the amount column into net_amount and tax_amount, a breaking change for anyone who sums amount.

In a centralized setup this change is a ticket to the central team, who discovers the 3 consumers by reading query logs and fixes them in 1 deploy, or misses one. In a mesh the orders team owns the change and the platform makes the safe path the easy one. The contract check in CI rejects the change as incompatible with version 1, so the team publishes a version 2 of the output port beside version 1. The catalog lists every consumer of version 1, and the owners hear about the change from the contract, not from a broken dashboard.

Both versions are served for 90 days. Each consumer moves on its own schedule, the supervision plane shows who still reads version 1, and the orders team retires it when that count reaches 0. The cost is running 2 versions for a while; what it buys is that no consumer breaks and no central team sits in the middle.

The 2 terms are often set against each other, but they answer different questions. Data mesh answers who owns data and how it is published: it's an operating model. Data fabric answers how teams find data spread across many systems and get integrated access to it: it's a technology layer built on metadata and automation, and it says nothing about who owns what.

That makes them compatible. A mesh needs discovery and access control that work across domains, and it needs those domains integrated. A fabric's tooling provides exactly that, so a mesh's self-serve platform can include fabric-style components. A company that buys a fabric while keeping 1 central team in the request path has improved access and left the queue where it was.

The data fabric guide covers the fabric's components in depth, and the data fabric vs data mesh comparison goes through the choice question by question.

A data lake is a place to store and query data, and so are a warehouse and a lakehouse. A data mesh is a way to divide ownership of that data. They're not alternatives, and a mesh does not require one physical store per domain.

Many meshes run on 1 shared lakehouse or warehouse. Each domain gets its own schemas or catalogs inside it, its own compute budget and its own permissions, while the table format and the catalog stay shared so products can be joined. Teams choose physical separation when compliance or scale demands it, or to allocate cost; the model doesn't require it.

What data mesh rejects is the central lake as an organization: 1 team that ingests everything into 1 place and becomes responsible for data it doesn't understand. Keeping the storage while moving the ownership is the most common way companies adopt it.

Data mesh pays off when the central team is the constraint and the domains can take on the work. The signs: many domains depend on 1 team, requests wait weeks, the central engineers keep modelling data wrong for lack of context, and domains are already building shadow pipelines. In that situation a mesh formalizes what's happening anyway and gives it standards.

It doesn't pay off when the queue is short. A company with a handful of domains and 1 data team that keeps up gains coordination overhead and platform cost and removes no bottleneck. The same holds when domains have no engineers who can own pipelines: decentralizing work to teams that can't do it moves the queue without shortening it.

Between the 2 sits the most common result: a central platform team, a few domains that own their products, and a central team that still serves the domains that aren't ready. Moving domains 1 at a time, starting where the queue hurts most, is how most successful programs grow.

Start with 1 or 2 domains and 1 real business problem, not with a company-wide domain map. Pick a domain whose data many others need, give it a named data product owner, and ship 1 product through a thin platform: a template, CI with contract checks, a shared table format and a catalog entry.

Grow the platform from what the first products needed, not from a design for all of them. Every step a domain had to do by hand is the next platform feature, and every rule the federation writes should arrive as a check in CI rather than a document. Add domains once the first product has consumers who rely on it.

The implementation patterns stay the same at any scale. Consumers find products in a data product catalog, and a schema registry or contract check stops producers from breaking them. A standard format lets products be joined, quality gates run before publication, and templates give each new product a working default to start from.

It requires skills most domains lack. Every owning domain needs engineers who can build and run pipelines. Hiring or training them costs more than growing 1 central team, and without them, decentralization is only a new name for the old queue.

Cross-domain work gets harder. Joining payments with marketing data now needs the 2 owners to agree on shared identifiers and schemas as well as freshness, and a question that spans 5 domains needs 5 of them to cooperate. Federated governance reduces this cost; it never removes it.

The platform is a permanent investment. It needs a strong team and a budget that outlasts the launch, because domains that find the platform harder than building their own will build their own. Duplication follows when discovery is weak: 2 domains derive the same metric in different ways because neither found the other's product.

These costs make data mesh expensive, so whether a company needs one usually comes down to its size, the depth of its domain teams and how long its central queue is.

Data mesh rarely appears as a definition question. It shows up in system design and senior loops as an ownership problem: who owns this dataset, how does a domain publish it, what happens when the producer ships a breaking change, and how would you know. Strong answers start from the bottleneck the model removes, name the trade-offs, and say when a central team is the better choice.

The follow-ups are concrete. Expect to design the pipeline a single domain runs, place the quality gate and the contract check, and explain how a versioned output port keeps consumers working. The data pipeline interview questions cover that ground as it's asked in real loops, and the data pipeline practice problems let you draw those designs on the same canvas an interview uses.

For the round itself, the data engineering system design round guide explains how interviewers scope the problem and what they expect at each level. A data catalog question often follows a mesh answer, since discovery is where most meshes succeed or fail.