Forward vs Backward Compatibility
Concepts covered: Schema Evolution
Writers and readers upgrade separately
A schema change splits the data into 2 versions: events written before it and events written after. It also splits the code that reads them, because the producer team and the data team ship on different days. For a while, every mix of old and new runs somewhere, and compatibility describes which mixes still work.
| Day | Producer writes | Loader code | Mix in play |
|---|---|---|---|
| Monday | v1 events, 4 fields | v1 | Old reader, old data |
| Tuesday | v2 events, adds `tip_usd` | v1 | Old reader, new data |
| Thursday | v2 events | v2, knows `tip_usd` | New reader, new data |
| Thursday, reprocessing last week | v1 events from the raw copy | v2 | New reader, old data |
Backward and forward
- Backward compatible
- New reader, old data
- The v2 loader reads v1 events
- Needed when new code reprocesses history
- Breaks when a new reader requires a field old events lack
- Forward compatible
- Old reader, new data
- The v1 loader reads v2 events
- Needed in the gap before readers upgrade
- Breaks when an old reader needs a field new events dropped
Name the direction from the reader's point of view. Backward means a new reader looks back at data from its past. Forward means an old reader receives data from its future. A change that keeps both working is fully compatible.
2 rules in the reading code
Compatibility comes down to 2 rules in the reading code. If the reader skips fields it doesn't recognize, new fields cannot hurt it. If every field it expects falls back to a default like null or 0 when missing, events that lack the field cannot hurt it. With both rules in place, the verdicts below follow.
| Change | Backward? | Forward? | Why |
|---|---|---|---|
| Add `tip_usd` with default 0 | Yes | Yes | New reader fills 0 for old events; old reader skips the field |
| Add `tip_usd` as required, no default | No | Yes | New reader finds no value in old events; old reader skips the field |
| Remove `promo_code`, which had a default | Yes | Yes | New reader skips it in old events; old reader fills the default |
| Remove `city`, required by old readers | Yes | No | New reader skips it in old events; old reader finds it missing |
| Rename `city` to `city_name` | No | No | Each side looks for a name the other side's events lack |
| Change `fare_usd` from a number to text | No | No | Neither reader expects the other's value type |
The trip service wants to add a required field vehicle_type with no default. Readers skip fields they do not recognize. How does the change classify?
- Forward compatible only: old readers skip it, but new readers fail on old events that lack it.
- Backward compatible only: new readers read old events, but old readers fail on the extra field.
- Fully compatible: adding a field never breaks a reader in either direction of time.
- Neither direction: any change to a schema breaks old readers and new readers alike.
On Tuesday the v1 loader meets v2 events, so the change must be forward compatible or Tuesday's load breaks. On Thursday the v2 loader rereads last week's v1 events, so it must be backward compatible or the reprocessing breaks. A batch pipeline that ever reprocesses history needs both, which is why a new field with a default is the change data teams hope to see.
On Monday the producer drops the optional field promo_code, whose default is null. The loader is upgraded on Friday. The next week, the team reruns the new loader over 30 days of raw events. Which direction was needed, and when?
- Backward from Monday to Friday, then forward during the 30-day rerun.
- Forward only, because once the loader is upgraded old events stop mattering.
- Forward from Monday to Friday, then backward during the 30-day rerun.
- Backward only, because the field was removed rather than added.
About this lesson
This section is part of the Schema Drift: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.