The Expand-Contract Pattern

Concepts covered: Schema Evolution

The registry blocks the rename, and the business still needs it. Today fare is a string such as 12.50 USD, and each of the topic's 3 consumers parses it with code of its own. The target is a long fare_cents plus a string fare_currency. Expand-contract makes that change in 4 deploys, each compatible with the one before it, so no team waits on another team's release.

One synchronized deploy
  • The producer and 3 consumers ship inside one release window
  • A rollback has to undo all 4 deploys together
  • A consumer that misses the window breaks on the first new message
  • Messages already on the topic still carry the old shape
Expand-contract
  • Each team ships on its own schedule
  • Each phase rolls back on its own
  • Old and new fields coexist for as long as readers need
  • The old field leaves only after its last reader has moved

The 4 phases

PhaseProducer writesConsumers readSchema
1. Expand`fare`, `fare_cents`, `fare_currency``fare`v3: adds 2 fields with null defaults, gives `fare` a default
2. Migrateall 3 fields`fare`, while history is backfilledv3, unchanged
3. Switchall 3 fieldseach consumer moves to `fare_cents` on its own datev3, unchanged
4. Contract`fare_cents`, `fare_currency``fare_cents`v4: deletes `fare`

Expand: add beside the old field

Version 3 adds fare_cents and fare_currency, both nullable with a default of null, and gives the existing fare a default of an empty string. Every change carries a default, so FULL passes. The producer now writes all 3 fields on every message. Nobody reads the new ones yet, so rolling back to version 2 breaks no reader. The default on fare does nothing today; it is what lets phase 4 delete the field.

Migrate: give history the new fields

The curated trips table holds 2 years of rows that only have fare. A backfill computes fare_cents and fare_currency for each trip_date partition and overwrites that partition, so a retried day produces the same rows. Before any consumer depends on the backfilled values, compare the backfill's parsing with what the producer wrote itself on recent days, where both columns exist.

SELECT
  trip_date,
  count(*) AS rows_checked,
  count_if(fare_cents <> CAST(round(CAST(split_part(fare, ' ', 1) AS DOUBLE) * 100) AS BIGINT)) AS mismatched_rows
FROM trips
WHERE trip_date >= DATE '2026-10-01'
GROUP BY trip_date
ORDER BY trip_date

During phase 2, the backfill for partition 2025-03-14 fails after writing half its rows, and the orchestrator retries the task. What makes the retry safe?

Switch: consumers move one by one

Each consumer team changes its reads to fare_cents, ships, and watches its own outputs. The earnings rollup might switch in week 1 and the fraud model in week 3, with the finance export last. Keep a written list of the topic's 3 consumers with a sign-off column: the registry records schemas and keeps no record of who reads which field.

Contract: remove what nobody reads

When the last sign-off lands, the producer ships version 4, which stops writing fare and deletes it from the schema. Because fare gained a default in phase 1, FULL accepts the deletion. Dropping the fare column from trips comes last, once a week passes with no query reading it.

Two of the 3 consumers of trip_completed read fare_cents. The third, the monthly finance export, still parses fare. The producer ships version 4, which stops writing fare. What does the finance export see on its next run?

About this lesson

This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.