Data Quality: Advanced

The version follows a major.minor.patch rule. An additive change, such as a new optional refund_reason field, raises it to 1.3.0 and asks nothing of consumers. A breaking change raises it to 2.0.0 and starts the 30-day notice. This team counts any edit to allowed as breaking, added values included, because a consumer that looks up exchange rates by currency fails on a code it has never seen. The amount_cents_meaning line matters as much as the types: a producer who starts including tax keeps every type valid and still moves every revenue figure downstream.

Continuous integration, or CI, is the set of automated checks that run on every proposed code change before it may merge. The contract check in the producer's CI builds a sample event from the changed code, usually inside a unit test, and diffs its fields against the contract file. If a pull request renames amount_cents to amount, the diff shows a required field missing and an undeclared field added. The check classifies that as breaking and fails the build unless the same pull request raises the contract to 2.0.0, which puts the rename in front of reviewers and starts the clock for consumers.

CI sees code, so it misses every problem the code did not cause. Suppose a payment partner starts routing orders through checkout with currency set to JPY by configuration. No code changed, so CI has nothing to check. The gate runs on each batch of real rows against the guarantees, so it catches the unknown currency, a day that lands 180,000 rows, or a replay that repeats order_id values. A failing gate quarantines the batch, and consumers keep reading the last good partition: the block authority from the beginner lesson, now driven by the contract.

Freshness is green everywhere, so the data arrived. Volume is normal in orders_raw and 31% low one step later, which places the loss in the step that builds orders_curated. That step inner joins orders to exchange rates on currency, and the distribution of orders_raw shows why rows vanish: a checkout release began sending lowercase eur, which matches no rate. Walking lineage downstream from orders_curated then finds a second victim, fraud_features, which no dashboard showed. A contract gate with a fixed currency list would have stopped this batch before orders_curated.

The incident passed through 2 gaps: no distribution check on orders_raw and no volume check on orders_curated. Either one would have fired hours before finance noticed. revenue_daily has 3 gaps of its own, so a fault that started there would surface only on the dashboard.

A commitment names an indicator and sets a target for it over a window. The operational indicator is direct: was the partition of orders_curated published by 06:00 UTC the next day? For quality, passing checks is a weak indicator, because a check says nothing about defects it never tests for. A stronger one comes from the corrections log, a record written whenever anyone restates a published partition. The quality indicator becomes: was the published partition never corrected? A partition the gate holds back costs nothing on quality, since no consumer read it.

A backtest replays a candidate threshold over stored history and counts how often it would have fired. The distributional checks you already run produce one value per metric per day; store them in quality.metric_history and the backtest becomes a query. This one scores each day's row count for orders_curated against the mean and standard deviation of the 8 previous same weekdays, since a Sunday should be compared with other Sundays. It tries 4 values of k, the number of standard deviations that triggers an alert, over the last 90 days.

Before annotations, only k = 4.0 fits the budget, and it misses 3 of the 4 incidents. Most extra alerts at lower settings fired on events the team already understood, and those events also stretched the baseline for the 8 weeks after them, which made later real drops look smaller.

After annotation, k = 2.5 catches all 4 incidents with 5 alerts, inside the budget. Give blocking checks a looser k than warning checks. A false warning costs an hour of reading. A false block holds a partition past 06:00 and spends the single spare day in the operational target.

On a new dataset the contract comes first and the data is built to match. A legacy table reverses that order. orders_legacy has 23 columns and has loaded nightly for 6 years. It has no written owner, and nobody has listed its readers. A clean contract enforced on day one would block tonight's load. The rollout runs in 4 steps, and each produces something the next one needs.

A breaking change cannot be planned without knowing whom it breaks. Query history lists who read the table and which columns they touched. A 90-day search found 13 readers; the 14th, an annual audit export that reads once a year, appeared only when the search covered 13 months. Of the 14, 9 are scheduled jobs and 3 are dashboards; the other 2 are analysts' notebooks. Of those, 5 filter on lowercase canceled only, so normalizing status in place would drop cancellations from their counts without any error. Record an owner and a contact for each consumer; step 4 needs them.

The cleanups go into contract v2.0.0 on a new table, orders_v2, under the same evolution policy as checkout's contract. There amount_cents becomes a bigint and status is normalized to 6 values. region is now required and filled from the delivery address. The legacy job writes both tables from the same input, each behind its own gate. Version 1 is frozen and receives fixes only. Each consumer from the inventory gets a dated move to v2, and orders_legacy is retired once every inventoried consumer has moved and the 30-day notice has passed.