Schema Registries: Where They Live
Concepts covered: Schema Evolution
A schema registry is a small service that stores every version of every event schema and checks each proposed version against the versions before it. In the beginner lesson, a producer changed its event shape and a consumer broke at runtime. A registry moves that failure to deploy time, where it costs a red build instead of a broken dashboard.
What the registry stores
Schemas are grouped by subject, usually 1 per topic, such as trip_completed-value. Each distinct schema receives a schema ID that is unique across the whole registry. A subject holds an ordered list of versions, and each version points to one of those IDs, so an identical schema registered under 2 subjects shares 1 ID. The subject also carries a compatibility mode: the rule every new version must pass before the registry accepts it.
{
"subject": "trip_completed-value",
"compatibility": "FULL",
"versions": [
{
"version": 1,
"schema_id": 412,
"fields": [
{"name": "trip_id", "type": "string"},
{"name": "driver_id", "type": "string"},
{"name": "fare", "type": "string"},
{"name": "ended_at", "type": "timestamp"}
]
},
{
"version": 2,
"schema_id": 587,
"fields": [
{"name": "trip_id", "type": "string"},
{"name": "driver_id", "type": "string"},
{"name": "fare", "type": "string"},
{"name": "ended_at", "type": "timestamp"},
{"name": "tip_cents", "type": "long", "default": 0}
]
}
]
}The registry also changes what travels on the topic. Each message carries a short schema ID ahead of its payload, in place of the full schema. A consumer that meets ID 587 for the first time fetches that schema from the registry once, caches it, and decodes every later message stamped 587 with it.
Deploy time and runtime
- trip_service (source, producer app)
- compat_check (quality gate, pre-deploy check); on error: fail the deploy
- schema_registry (storage, registry service)
- trip_completed (queue, event topic)
- earnings_rollup (consumer, streaming job)
- fraud_scoring (consumer, streaming job)
- finance_export (consumer, monthly batch)
- trip_service -> compat_check
- compat_check -> schema_registry
- trip_service -> trip_completed
- trip_completed -> earnings_rollup
- trip_completed -> fraud_scoring
- trip_completed -> finance_export
- schema_registry -> earnings_rollup
- schema_registry -> fraud_scoring
- schema_registry -> finance_export
The check that matters runs before the deploy. When trip_service is about to ship a new event shape, a build step submits the proposed schema to the registry and asks whether it passes the subject's compatibility mode. A failure stops the release, so the incompatible shape never produces a message. Runtime is the backstop: a producer that skips the check and has auto-registration turned on tries to register the new version when it first serializes a message, and that send fails with a compatibility error where it expected a schema ID.
The 4 compatibility modes
| Mode | Guarantee | Passes | Fails |
|---|---|---|---|
| BACKWARD | Readers on the new schema decode old messages | Delete a field; add a field with a default | Add a field with no default |
| FORWARD | Readers on the old schema decode new messages | Add a field; delete a field that has a default | Delete a field with no default |
| FULL | Both directions hold | Add or delete fields that have defaults | Add or delete a field with no default |
| NONE | No guarantee | Any change | Nothing |
The trip_completed-value subject uses FORWARD compatibility. A proposed version 3 deletes driver_id, a required string with no default in version 2. What does the check return?
- It passes, because FORWARD only inspects the fields a new version adds.
- It fails, because readers still on version 2 need driver_id and have no default.
- It passes, and readers still on version 2 get null wherever driver_id was.
- It fails, because FORWARD rejects every deletion, including fields that have defaults.
The mode also fixes who deploys first. Under BACKWARD, consumers upgrade first, since a new reader can decode old messages. Under FORWARD, the producer upgrades first, since old readers can decode new messages. FULL lets either side go first, which suits a topic read by several teams that ship on separate schedules.
Each mode compares a candidate with the latest version only. The transitive variants, such as FULL_TRANSITIVE, compare it with every registered version. Choose a transitive mode when consumers replay history: a topic that retains 30 days can hold messages written under 4 versions, and a replay from day 1 decodes all of them.
Which proposed version 3 change passes under FULL compatibility?
- Add pickup_zone as an optional string with default 'unknown'.
- Change fare from a string to a long that holds cents.
- Add vehicle_type as a required string that has no default.
- Rename driver_id to partner_id and keep the same type.
About this lesson
This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.