Schema Registries and Contracts

Concepts covered: Schema Evolution

A registry checks shape

A schema registry stores every version of a schema under a subject, usually 1 subject per stream or table, assigns each version an ID, and refuses any new version that fails the subject's compatibility mode. Producers prefix each message with the ID of the schema that wrote it, a few bytes, so a consumer can fetch the exact writer schema and resolve it against its own reader schema. The check runs at registration time, which makes it a gate in the producer's build: an incompatible schema fails before any event in that shape exists.

The build registers the schema before deploy. Each event carries its schema ID, and the gate fetches that schema from the registry before checking the event against the contract.

producer_ci (transform, Producer build pipeline); on error: fail build on incompatible schema
schema_registry (storage, Schema registry, transitive backward mode)
checkout_service (source, Checkout service)
orders_stream (queue, Event stream, 7-day retention)
raw_orders (bronze, Raw table, JSON payload)
ingest_gate (quality gate, Contract checks per event); on error: route to dead-letter table; alerting: page if over 0.5% fail in 10 min
dead_letter (storage, Dead-letter table); backfill: replay through gate after fix
stg_orders (silver, Typed staging table); idempotency: merge on order_id

producer_ci -> schema_registry
checkout_service -> orders_stream
orders_stream -> raw_orders
raw_orders -> ingest_gate
schema_registry -> ingest_gate
ingest_gate -> stg_orders
ingest_gate -> dead_letter

Registry catches

A field removed that had no default
A type changed from bigint to string
A rename with no alias
A new field with no default, under backward mode

Registry passes, contract catches

amount_cents switching from US cents to minor units
status gaining a new value, PARTIAL_REFUND
created_at switching from UTC to local time
customer_id going from 0% to 30% null

Everything in the right column is a valid value of a valid type. The registry's work ends at shape, and the Japan plan would have passed it on the first try.

A contract adds meaning and an owner

A data contract is the producer's written promise about a dataset: the schema, what each field means and in which unit, and the constraints values obey. It also names the owner and the consumers, and it says how changes will be announced. The schema part can live in the registry. The rest needs a file the producer reviews like code, plus checks that read it.

contract: orders
contract_version: 1
owner: payments-platform
subject: orders-value
compatibility: backward_transitive
freshness_sla_minutes: 15
fields:
  order_id:
    type: string
    required: true
    unique: true
  customer_id:
    type: bigint
    required: true
  amount_cents:
    type: bigint
    required: true
    unit: USD cents
    min: 0
  created_at:
    type: timestamp
    timezone: UTC
    required: true
change_policy:
  additive: notify consumers 7 days ahead
  semantic_or_breaking: new field, 30-day overlap
consumers:
  - finance-revenue
  - fraud-scoring
  - partner-export

No tool reads intent, so the contract works through people and rules. With unit: USD cents on the page, the payments engineer adding currency reads that line in review and has to decide what yen orders put in amount_cents, which is the question the Japan plan skipped. The change policy classes the redefinition as semantic, so it ships as a new field with a 30-day overlap rather than as a reinterpretation of an old one. The gate can also enforce a rule derived from the contract: while amount_cents is US cents, an event whose currency is not USD fails.

Ownership decides who can change what. The producer owns the contract and is the only party that publishes a new version. Consumers add themselves to the consumers list, which makes that list the notification list for every change and the evidence needed before anything is dropped.

Halting or quarantining at the gate

Check point	What it checks	Cost of a failure there
Producer build	Registry compatibility and a diff of the contract file	A blocked merge; no bad event exists
Ingestion gate	Per-event constraints: nulls, ranges, allowed values, units	Events wait in the dead-letter table and curated data lags
Consumer tests	Assumptions one consumer relies on, run against staging	A failed deploy of that consumer
Daily reconciliation	Curated totals against the source system	Wrong numbers already published for a day

The earlier a check runs, the cheaper its failure. At the gate, the choice is between halting and flowing. Blocking the whole stream on one bad event stops the revenue fact for every consumer; routing bad events to a dead-letter table keeps it flowing but leaves it quietly incomplete. The usual answer is per-event quarantine plus a rate alarm: page when more than 0.5% of events fail within 10 minutes, since a failure rate that high almost always traces to a producer deploy.

Do

Give every contract a named owning team and a list of consumers
Write the unit and timezone of every numeric and time field
Run compatibility checks in the producer's build, before deploy
Replay dead-lettered events through the same gate after a fix

Avoid

Treat a passing registry check as proof that a change is safe
Let consumers register schemas for data they do not produce
Halt a shared stream because one event fails validation
Write contracts for every table at once instead of starting with money and shared sources

About this lesson

This section is part of the Schema Evolution: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.