Adding Is Safe, Renaming Is Not
Concepts covered: Schema Evolution
3 classes of schema change
The compatibility test judges a single change. A proposal also needs a verdict on its plan: can it ship today, or does it need a schedule agreed with other teams? Each proposal falls into 1 of 3 classes.
- Additive: A new optional field with a default, or a new table. Readers keep working, so the producer ships without waiting for anyone.
- Destructive: Any single release that drops a field or changes its name or type. Some reader breaks the moment it ships, either loudly with an error or quietly with wrong numbers.
- Coordinated: The same rename or type change, split so that no step removes anything a reader still uses. It takes weeks and needs every reader's owner to agree.
A rename looks like a delete to every reader
A rename feels small to the team making it. To every reader it is 2 changes at once: a field they rely on disappears, and an unknown field appears. Suppose the trip service adopts a naming standard and renames fare_usd to fare_amount_usd in a release at 15:00 on March 10. The loader runs in ignore mode, so it skips the unknown name and stores null in fare_usd. Every load succeeds.
SELECT
date(completed_at) AS trip_date,
sum(fare_usd) AS revenue_usd
FROM trips
WHERE date(completed_at) BETWEEN DATE '2026-03-08' AND DATE '2026-03-11'
GROUP BY date(completed_at)
ORDER BY trip_date| trip_date | revenue_usd |
|---|---|
| 2026-03-08 | 612,400 |
| 2026-03-09 | 598,900 |
| 2026-03-10 | 214,700 |
| 2026-03-11 | null |
March 10 holds only the fares from before 15:00, and March 11 sums nothing but nulls, which is null. Revenue appears to fall by almost two thirds and then vanish. No run fails, and a finance analyst is the first to notice. A reject loader would stop on March 10 instead. That is the better failure because it is louder, but readers break either way, so a single-step rename is destructive.
Which of these proposals is coordinated?
- Add an optional surge_multiplier field with a default of 1.0 in Thursday's release.
- Remove promo_code in Thursday's release, since the app no longer shows promo codes.
- Change completed_at from a timestamp to text in Thursday's release, keeping its name.
- Add rider_rating, copy values from rating, move readers over, then drop rating.
The Safe Rename: Add, Copy, Switch, Drop
A coordinated rename reaches the same end state in 4 steps, and nothing a reader uses disappears until that reader has stopped using it.
| Step | Producer writes | Readers use | Why nothing breaks |
|---|---|---|---|
| 1. Add | Both `fare_usd` and `fare_amount_usd` | `fare_usd` | A new field passes both compatibility directions |
| 2. Copy | Both | `fare_usd` | Rows from before step 1 get `fare_amount_usd` filled from `fare_usd` |
| 3. Switch | Both | `fare_amount_usd`, a reader at a time | Both names hold the same values, so each reader moves on its own schedule |
| 4. Drop | `fare_amount_usd` only | `fare_amount_usd` | No reader is left on the old name |
SELECT
count(*) AS rows_missing_new_name
FROM trips
WHERE fare_amount_usd IS NULL
AND fare_usd IS NOT NULLRun the steps out of order and readers break. Drop before switch, and any reader still on fare_usd reads nulls: the March 11 dashboard again. Switch before the copy finishes, and a switched reader sees nulls for every row older than step 1. A type change uses the same steps under a new name: to store fares as whole cents, add fare_cents, copy, switch, then drop the old field.
- Do
- Ship new fields with a default so they pass both directions
- Treat every rename as add, copy, switch, drop
- List every reader of a field before removing it
- Avoid
- Rename or retype a field in a single release
- Drop the old field because the producer stopped writing it
- Trust ignore mode during a rename: it hides the break
A team renaming city to city_name finishes the add and copy steps, then removes city from the producer and the table the next morning. What happens?
- Nothing breaks, because the copy step already put every value under the new name.
- Every reader that still selects city breaks, because the switch step was skipped.
- Only new events break, because old rows in the table still hold their city values.
- Readers break only if the loader is in reject mode, because ignore mode shields them.
About this lesson
This section is part of the Schema Drift: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.