Schema Registries: Where They Live

Concepts covered: Schema Evolution

A schema registry is a small service that stores every version of every event schema and checks each proposed version against the versions before it. In the beginner lesson, a producer changed its event shape and a consumer broke at runtime. A registry moves that failure to deploy time, where it costs a red build instead of a broken dashboard.

What the registry stores

Schemas are grouped by subject, usually 1 per topic, such as trip_completed-value. Each distinct schema receives a schema ID that is unique across the whole registry. A subject holds an ordered list of versions, and each version points to one of those IDs, so an identical schema registered under 2 subjects shares 1 ID. The subject also carries a compatibility mode: the rule every new version must pass before the registry accepts it.

{
  "subject": "trip_completed-value",
  "compatibility": "FULL",
  "versions": [
    {
      "version": 1,
      "schema_id": 412,
      "fields": [
        {"name": "trip_id", "type": "string"},
        {"name": "driver_id", "type": "string"},
        {"name": "fare", "type": "string"},
        {"name": "ended_at", "type": "timestamp"}
      ]
    },
    {
      "version": 2,
      "schema_id": 587,
      "fields": [
        {"name": "trip_id", "type": "string"},
        {"name": "driver_id", "type": "string"},
        {"name": "fare", "type": "string"},
        {"name": "ended_at", "type": "timestamp"},
        {"name": "tip_cents", "type": "long", "default": 0}
      ]
    }
  ]
}

The registry also changes what travels on the topic. Each message carries a short schema ID ahead of its payload, in place of the full schema. A consumer that meets ID 587 for the first time fetches that schema from the registry once, caches it, and decodes every later message stamped 587 with it.

Deploy time and runtime

Before a deploy, the check asks the registry whether the proposed schema is compatible. At runtime, trip_service stamps each message with a schema ID, and each consumer fetches the matching schema from the registry.
  • trip_service (source, producer app)
  • compat_check (quality gate, pre-deploy check); on error: fail the deploy
  • schema_registry (storage, registry service)
  • trip_completed (queue, event topic)
  • earnings_rollup (consumer, streaming job)
  • fraud_scoring (consumer, streaming job)
  • finance_export (consumer, monthly batch)
  • trip_service -> compat_check
  • compat_check -> schema_registry
  • trip_service -> trip_completed
  • trip_completed -> earnings_rollup
  • trip_completed -> fraud_scoring
  • trip_completed -> finance_export
  • schema_registry -> earnings_rollup
  • schema_registry -> fraud_scoring
  • schema_registry -> finance_export

The check that matters runs before the deploy. When trip_service is about to ship a new event shape, a build step submits the proposed schema to the registry and asks whether it passes the subject's compatibility mode. A failure stops the release, so the incompatible shape never produces a message. Runtime is the backstop: a producer that skips the check and has auto-registration turned on tries to register the new version when it first serializes a message, and that send fails with a compatibility error where it expected a schema ID.

The 4 compatibility modes

ModeGuaranteePassesFails
BACKWARDReaders on the new schema decode old messagesDelete a field; add a field with a defaultAdd a field with no default
FORWARDReaders on the old schema decode new messagesAdd a field; delete a field that has a defaultDelete a field with no default
FULLBoth directions holdAdd or delete fields that have defaultsAdd or delete a field with no default
NONENo guaranteeAny changeNothing

The trip_completed-value subject uses FORWARD compatibility. A proposed version 3 deletes driver_id, a required string with no default in version 2. What does the check return?

The mode also fixes who deploys first. Under BACKWARD, consumers upgrade first, since a new reader can decode old messages. Under FORWARD, the producer upgrades first, since old readers can decode new messages. FULL lets either side go first, which suits a topic read by several teams that ship on separate schedules.

Each mode compares a candidate with the latest version only. The transitive variants, such as FULL_TRANSITIVE, compare it with every registered version. Choose a transitive mode when consumers replay history: a topic that retains 30 days can hold messages written under 4 versions, and a replay from day 1 decodes all of them.

Which proposed version 3 change passes under FULL compatibility?

About this lesson

This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.