Five Pillars of Observability

Concepts covered: Data Quality & Validation

Barr Moses and the Monte Carlo Data team named the five pillars of data observability: freshness, distribution, volume, schema, and lineage. The naming has caught on widely enough that conversations about quality use it as shorthand. The pillars are useful because they are not a checklist; they are a diagnostic framework. When something is wrong with the data, the pillar that detected the symptom narrows the search for the cause. When designing a quality program, the pillars name the gaps that have to be filled before the program is considered observable rather than instrumented alone. The framework predates the pillars under different names. Software observability matured along the same lines: metrics, logs, and traces are a similar three-axis decomposition, where each axis answers a different diagnostic question. The data version is more axes because data has more independent dimensions of failure. Volume can be wrong without distribution being wrong; schema can shift without freshness slipping; lineage can be unknown even when every other pillar is green.

The Five Pillars

Freshness: Time between latest available record and now. Asks whether the pipeline is keeping up. The simplest pillar to measure and the first one consumers notice.
Distribution: Statistical properties of columns: mean, stddev, quantiles, cardinality, category mix. Catches shifts that no individual row violates.
Volume: Row counts compared to historical baselines. Catches dropped partitions, broken filters, exploded joins. Often the first signal of an upstream problem.
Schema: Columns, types, nullability, accepted values, ranges. The producer-side commitment made executable.
Lineage: Upstream and downstream relationships at column granularity. Turns 'a number changed' into 'a number changed because this transform changed'.

Pillars As A Diagnostic Framework

When a consumer reports a wrong number, a senior engineer walks the pillars in order and uses each one to either narrow or rule out a class of cause. Freshness rules out 'is the data even recent.' Volume rules out 'did the right amount of data arrive.' Schema rules out 'is the shape correct.' Distribution rules out 'are the values within their normal range.' Lineage answers 'what produced this column and what depends on it.' The walk takes minutes and replaces the unstructured 'try things until something works' debugging that consumes hours when each pillar has to be checked manually.

Symptom	First Pillar To Check	Why
Dashboard shows last week's data on Monday morning	Freshness	Most likely an ingestion stall; rules in or out the simplest cause
Revenue dropped 15 percent overnight	Volume	Sudden numeric drops are usually missing rows, not changed values
Average order amount is up but row count is steady	Distribution	Aggregate change without row count change implies value shift
Pipeline succeeded but downstream join produces nulls	Schema	Type or nullability mismatch is the typical cause of post-success join failures
Two consumers see different numbers from the same source	Lineage	Different consumers may read different downstream tables; lineage exposes the divergence

A pillar is a category. A check is a specific assertion within a pillar. The pillar 'distribution' contains many checks: mean shift, stddev shift, p99 shift, category mix shift, cardinality shift. The framework value of pillars is in coverage, not in implementation. A program that has a hundred checks within four pillars and zero checks in the fifth is a program with a known blind spot.

Lineage: The Pillar Most Programs Skip

Most quality programs cover four pillars well and lineage poorly. The reason is cost. Lineage at table granularity is moderately expensive; lineage at column granularity is expensive; lineage that updates as transforms evolve is expensive to keep current. The payoff is that lineage transforms incident response. A column-level lineage system answers questions like 'what consumers depend on amount_usd in fct_orders' in seconds. Without lineage, the same question takes hours of grep-and-Slack archaeology. Tools like dbt, Dagster, and standalone catalogs (DataHub, OpenMetadata) cover this pillar; the discipline is in adopting and keeping them current.

Column-level lineage example for fct_orders.amount_usd:

  raw.events.amount  ->  curated.fct_orders.amount_usd  -+-> mart.daily_revenue.revenue_usd
                                                         |                                  
                                                         +-> feature_store.user_features.lifetime_spend
                                                         |                                  
                                                         +-> reverse_etl.salesforce.account_revenue

Reading the diagram backward from a consumer answers 'where does this number come from.' Reading it forward from a source column answers 'who is affected if this column changes.' Both questions show up in incident response and in change reviews. Lineage is the pillar that ties the other four together.

When To Adopt Each Pillar

Maturity Stage	Pillars In Place	What The Team Can Answer
Stage 1: Cheap checks	Volume, freshness	Did the right amount of data arrive on time
Stage 2: Suite	Volume, freshness, schema, distribution	Is the data structurally and statistically as expected
Stage 3: Observable	All five pillars including lineage	What changed, why, and who is affected
Stage 4: Contract-enforced	All five pillars plus contracts in CI	Same as Stage 3, but most failures cannot ship

Without Lineage

Incident response starts with 'who owns this column'
Change reviews miss downstream consumers
Deprecation requires manual canvassing
Two consumers compute conflicting numbers; cause is hidden

With Lineage

Incident response starts with the lineage graph; ownership is metadata
Change reviews automatically surface affected consumers
Deprecation walks the graph and notifies every downstream
Conflicting numbers are explained by divergent transforms in the graph

5 Pillars cover the diagnostic surface; fewer leaves a known blind spot: Freshness, distribution, volume, schema, lineage

The five pillars are a framework for coverage and a diagnostic walk for incidents.
Most programs underinvest in lineage; the cost shows up in incident response time.
Pillars are categories; specific checks live within them. Coverage means hitting all five categories, not running one check.

The pillars are descriptive, not prescriptive. A program with twenty schema checks and zero distribution checks is not 'four-fifths observable'; it is observable on schema and blind on distribution. Coverage is binary per pillar.

About this lesson

This section is part of the Data Quality: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.