Forward vs Backward Compatibility

Concepts covered: Schema Evolution

Writers and readers upgrade separately

A schema change splits the data into 2 versions: events written before it and events written after. It also splits the code that reads them, because the producer team and the data team ship on different days. For a while, every mix of old and new runs somewhere, and compatibility describes which mixes still work.

DayProducer writesLoader codeMix in play
Mondayv1 events, 4 fieldsv1Old reader, old data
Tuesdayv2 events, adds `tip_usd`v1Old reader, new data
Thursdayv2 eventsv2, knows `tip_usd`New reader, new data
Thursday, reprocessing last weekv1 events from the raw copyv2New reader, old data

Backward and forward

Backward compatible
  • New reader, old data
  • The v2 loader reads v1 events
  • Needed when new code reprocesses history
  • Breaks when a new reader requires a field old events lack
Forward compatible
  • Old reader, new data
  • The v1 loader reads v2 events
  • Needed in the gap before readers upgrade
  • Breaks when an old reader needs a field new events dropped

Name the direction from the reader's point of view. Backward means a new reader looks back at data from its past. Forward means an old reader receives data from its future. A change that keeps both working is fully compatible.

2 rules in the reading code

Compatibility comes down to 2 rules in the reading code. If the reader skips fields it doesn't recognize, new fields cannot hurt it. If every field it expects falls back to a default like null or 0 when missing, events that lack the field cannot hurt it. With both rules in place, the verdicts below follow.

ChangeBackward?Forward?Why
Add `tip_usd` with default 0YesYesNew reader fills 0 for old events; old reader skips the field
Add `tip_usd` as required, no defaultNoYesNew reader finds no value in old events; old reader skips the field
Remove `promo_code`, which had a defaultYesYesNew reader skips it in old events; old reader fills the default
Remove `city`, required by old readersYesNoNew reader skips it in old events; old reader finds it missing
Rename `city` to `city_name`NoNoEach side looks for a name the other side's events lack
Change `fare_usd` from a number to textNoNoNeither reader expects the other's value type

The trip service wants to add a required field vehicle_type with no default. Readers skip fields they do not recognize. How does the change classify?

On Tuesday the v1 loader meets v2 events, so the change must be forward compatible or Tuesday's load breaks. On Thursday the v2 loader rereads last week's v1 events, so it must be backward compatible or the reprocessing breaks. A batch pipeline that ever reprocesses history needs both, which is why a new field with a default is the change data teams hope to see.

On Monday the producer drops the optional field promo_code, whose default is null. The loader is upgraded on Friday. The next week, the team reruns the new loader over 30 days of raw events. Which direction was needed, and when?

About this lesson

This section is part of the Schema Drift: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.