Handling Breaking Changes

Concepts covered: Schema Evolution

Expand, migrate, contract

The fix agreed with finance: yen sales wait, and amount_cents is replaced by amount_minor, in each currency's own minor unit, alongside the currency field that v2 already added. Removing and redefining a field breaks both directions, so no single version can ship it. A sequence of versions can, each compatible with the one before, ending where the breaking change would have.

The hard decision comes in the migrate phase. Once yen orders exist, what does amount_cents hold for them? Leaving it null makes old readers undercount Japan, and copying the yen amount repeats the original bug. The producer fills it with the US-cent equivalent at the booking-time exchange rate, which keeps the old field true to its contract until the last old reader is gone.

The contract phase needs evidence before it starts: query logs or access audits showing that nothing has selected amount_cents for 30 days. A consumer missing from the contract's list is the usual reason a drop breaks someone, which is why consumers maintain that list themselves.

A view as a compatibility shim

The partner cannot change its parser inside 30 days, so the curated layer serves it a view that rebuilds the old field from the new pair, using 2 reference tables.

CREATE OR REPLACE VIEW curated.orders_legacy AS
SELECT
  o.order_id,
  o.customer_id,
  CASE
    WHEN o.currency = 'USD' THEN o.amount_minor
    ELSE CAST(round(o.amount_minor * r.usd_per_unit * power(10, 2 - c.minor_unit_exponent)) AS bigint)
  END AS amount_cents,
  o.created_at
FROM curated.orders AS o
JOIN reference.currencies AS c
  ON o.currency = c.currency_code
LEFT JOIN reference.fx_daily AS r
  ON o.currency = r.currency_code
  AND CAST(o.created_at AS date) = r.rate_date

The exponent matters because amount_minor is in each currency's own minor unit: yen has 0 decimal places, the dollar 2. Dollar orders pass amount_minor through unchanged, so they never depend on a rate. The join to fx_daily is a LEFT JOIN on purpose. Rates for a day can land hours after that day's orders, and an inner join would drop those orders from the export without a trace. With the left join they appear with a null amount, and because a view is computed at read time, the next export after the rate lands is correct with no reprocessing. A materialized table built from the same query would keep the nulls until someone reruns it.

Old shapes keep arriving

Contracting the schema does not retire the old shapes. A consumer recovering from an outage replays up to 7 days of the stream, reaching back past the checkout service's latest deploy. Events quarantined during a rollout leave the dead-letter table weeks later in the shape they were written in. The raw table holds 3 years of v1 and v2 payloads that any backfill reads. All of it is late data in an old shape, so the staging reader keeps upcasters: functions that each turn one version into the next, chained until the event reaches the latest.

def v1_to_v2(event: dict) -> dict:
    return {**event, "currency": "USD", "schema_version": 2}


def v2_to_v3(event: dict) -> dict:
    return {**event, "amount_minor": event["amount_cents"], "schema_version": 3}


def v3_to_v4(event: dict) -> dict:
    upgraded = {k: v for k, v in event.items() if k != "amount_cents"}
    upgraded["schema_version"] = 4
    return upgraded


UPCASTERS = {1: v1_to_v2, 2: v2_to_v3, 3: v3_to_v4}
LATEST_VERSION = 4


def to_latest(event: dict) -> dict:
    version = event.get("schema_version", 1)
    while version < LATEST_VERSION:
        event = UPCASTERS[version](event)
        version = event["schema_version"]
    return event

Each step rests on a fact about its version. The checkout service wrote every v1 and v2 event before it accepted any non-USD order, since yen sales open only with v3, so every one of them is in US cents and copying amount_cents into amount_minor is exact. Upcasters are deterministic, so a replay produces identical rows and the staging merge on order_id absorbs the duplicates a retry creates. An upcaster can be deleted only when no retained store holds its version. With 3 years of raw history, that means keeping it for 3 years or rewriting the raw table, which costs a full scan and cannot be undone; a 3-line function is usually the cheaper of the two.

Evolve in place

• One stream and one table to reason about
• Each field migrates on its own timeline
• Overlap lives inside one schema, as extra fields
• Fits field-level changes: renames, units, types

Publish orders_v2 alongside

• Clean break, with the old stream frozen as it was
• Producer writes twice and storage doubles during the overlap
• Consumers switch endpoints, and ordering across the 2 streams is lost
• Fits changes of grain or key, such as orders becoming order lines

About this lesson

This section is part of the Schema Evolution: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.