Forward and Backward Compatibility

Concepts covered: Schema Evolution

Writer schema, reader schema

Every compatibility statement compares 2 schemas: the one that wrote the data and the one reading it. Backward compatible means code on the new schema can read data written with the old one. Forward compatible means code still on the old schema can read data written with the new one. Full compatibility means both hold. The names get swapped under pressure, so tie each to a deployment: backward compatibility lets readers upgrade first, forward compatibility lets writers upgrade first.

Backward: readers upgrade first

• Readers on the new schema meet old data during rollout
• Fields the old data lacks need a default
• Lets a field be removed: readers stop using it, then writers stop sending it
• Fits lake tables whose old files stay readable for years

Forward: writers upgrade first

• Readers on the old schema meet new data during rollout
• Old readers must skip fields they do not know
• Lets a field be added: writers send it before any reader uses it
• Fits streams whose consumers deploy on their own schedules

Change by change

Two assumptions sit under every Yes in that table. The reader resolves fields by name, so a field it does not list is skipped, and the format can promote int to bigint while reading. A positional reader breaks the first assumption, which is why, for the partner's positional CSV export, almost no change is forward compatible: only a field appended at the end leaves the old positions intact.

Rollout order follows from the verdict. Adding currency with a default is fully compatible, and the producer can deploy before or after its consumers. Removing a field with no default is backward only: every consumer must stop reading it and redeploy before the producer stops writing it, and the producer's deploy waits on the slowest consumer.

Formats differ in details, such as which numeric promotions they allow and whether a field can carry aliases for renames; the 2 directions are the same in all of them.

A default can hide a missing currency

Resolution by name with defaults fits in a short function, and writing it out shows where the judgement sits.

from typing import Any, Dict

READER_V2 = {
    "order_id": {"type": str},
    "customer_id": {"type": int},
    "amount_cents": {"type": int},
    "created_at": {"type": str},
    "currency": {"type": str, "default": "USD"},
}


def resolve(record: Dict[str, Any], reader: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
    resolved = {}
    for name, spec in reader.items():
        if name in record:
            value = record[name]
        elif "default" in spec:
            value = spec["default"]
        else:
            raise ValueError(f"missing required field: {name}")
        if not isinstance(value, spec["type"]):
            raise TypeError(f"{name} should be {spec['type'].__name__}")
        resolved[name] = value
    return resolved


v1_event = {
    "order_id": "o-1001",
    "customer_id": 77,
    "amount_cents": 4599,
    "created_at": "2026-10-01T09:14:00Z",
    "app_build": "8.4.1",
}
print(resolve(v1_event, READER_V2))

The v1 event comes back with currency set to USD, and app_build, which the reader does not list, is dropped. The USD default is correct for every order placed before the Japan launch, and a hazard for every writer after it: a new mobile build that forgets to set currency on a yen order also resolves to USD, and the schema has hidden the bug. The stricter design gives currency a null default. That keeps compatibility while leaving missing values visible, and staging makes the decision. A v1 event predates the field, so it becomes USD; a v2 event with a null currency goes to the dead-letter table.

Compatible with which version?

A registry can check a new schema against the latest version only, or against every version it holds, which is called transitive checking. The difference shows up wherever old data stays readable. Take a members event, separate from orders:

Each version passed the check it faced. A backfill that reads 3 years of raw member events, most of them written as v1, with the v3 reader meets a bigint where it expects a string and stops. A stream that keeps 7 days of events can often live with checks against the previous version, because nothing older is read from it. Any store holding history needs transitive backward compatibility.

About this lesson

This section is part of the Schema Evolution: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.