The Expand-Contract Pattern
Concepts covered: Schema Evolution
The registry blocks the rename, and the business still needs it. Today fare is a string such as 12.50 USD, and each of the topic's 3 consumers parses it with code of its own. The target is a long fare_cents plus a string fare_currency. Expand-contract makes that change in 4 deploys, each compatible with the one before it, so no team waits on another team's release.
- One synchronized deploy
- The producer and 3 consumers ship inside one release window
- A rollback has to undo all 4 deploys together
- A consumer that misses the window breaks on the first new message
- Messages already on the topic still carry the old shape
- Expand-contract
- Each team ships on its own schedule
- Each phase rolls back on its own
- Old and new fields coexist for as long as readers need
- The old field leaves only after its last reader has moved
The 4 phases
| Phase | Producer writes | Consumers read | Schema |
|---|---|---|---|
| 1. Expand | `fare`, `fare_cents`, `fare_currency` | `fare` | v3: adds 2 fields with null defaults, gives `fare` a default |
| 2. Migrate | all 3 fields | `fare`, while history is backfilled | v3, unchanged |
| 3. Switch | all 3 fields | each consumer moves to `fare_cents` on its own date | v3, unchanged |
| 4. Contract | `fare_cents`, `fare_currency` | `fare_cents` | v4: deletes `fare` |
Expand: add beside the old field
Version 3 adds fare_cents and fare_currency, both nullable with a default of null, and gives the existing fare a default of an empty string. Every change carries a default, so FULL passes. The producer now writes all 3 fields on every message. Nobody reads the new ones yet, so rolling back to version 2 breaks no reader. The default on fare does nothing today; it is what lets phase 4 delete the field.
Migrate: give history the new fields
The curated trips table holds 2 years of rows that only have fare. A backfill computes fare_cents and fare_currency for each trip_date partition and overwrites that partition, so a retried day produces the same rows. Before any consumer depends on the backfilled values, compare the backfill's parsing with what the producer wrote itself on recent days, where both columns exist.
SELECT
trip_date,
count(*) AS rows_checked,
count_if(fare_cents <> CAST(round(CAST(split_part(fare, ' ', 1) AS DOUBLE) * 100) AS BIGINT)) AS mismatched_rows
FROM trips
WHERE trip_date >= DATE '2026-10-01'
GROUP BY trip_date
ORDER BY trip_dateDuring phase 2, the backfill for partition 2025-03-14 fails after writing half its rows, and the orchestrator retries the task. What makes the retry safe?
- The registry rejects the duplicate rows, since they carry the same schema ID.
- The backfill only writes fare_cents, so duplicates in other columns do no harm.
- FULL compatibility guarantees each message is decoded exactly once by readers.
- Each run overwrites the whole partition, so the retry replaces the partial write.
Switch: consumers move one by one
Each consumer team changes its reads to fare_cents, ships, and watches its own outputs. The earnings rollup might switch in week 1 and the fraud model in week 3, with the finance export last. Keep a written list of the topic's 3 consumers with a sign-off column: the registry records schemas and keeps no record of who reads which field.
Contract: remove what nobody reads
When the last sign-off lands, the producer ships version 4, which stops writing fare and deletes it from the schema. Because fare gained a default in phase 1, FULL accepts the deletion. Dropping the fare column from trips comes last, once a week passes with no query reading it.
Two of the 3 consumers of trip_completed read fare_cents. The third, the monthly finance export, still parses fare. The producer ships version 4, which stops writing fare. What does the finance export see on its next run?
- A schema error, because the registry blocked version 4 for breaking an old reader.
- A decode failure on every message, because the fare field is missing.
- Messages decode, fare arrives as the empty-string default, and the parser finds no amount.
- Correct fares, because the reader rebuilds fare from fare_cents during decoding.
About this lesson
This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.