Schema Evolution Without Rewrites

Columnar files record each column's name inside the file. Older lake engines matched file columns to the table schema by that name or by position, and both break. Rename amt to amount on a name-matching table and every file written before the rename lacks an amount column, so 3 years of orders read back as NULL. On a position-matching table, dropping the second column shifts every later column one slot. The traditional fix was to rewrite every file.

Field ids name the column

A table format assigns each column a numeric field id when the column is created and never reuses it. Writers store the id beside each column in every data file, and readers resolve columns by id, so the name becomes a label in metadata that can change freely. Here is the schema history in the root metadata of orders:

{
  "current_schema_id": 2,
  "schemas": [
    {
      "schema_id": 1,
      "fields": [
        {"id": 1, "name": "order_id", "type": "bigint"},
        {"id": 2, "name": "order_date", "type": "date"},
        {"id": 3, "name": "amt", "type": "integer"},
        {"id": 4, "name": "legacy_flag", "type": "boolean"}
      ]
    },
    {
      "schema_id": 2,
      "fields": [
        {"id": 1, "name": "order_id", "type": "bigint"},
        {"id": 2, "name": "order_date", "type": "date"},
        {"id": 3, "name": "amount", "type": "bigint"},
        {"id": 5, "name": "currency", "type": "varchar"}
      ]
    }
  ]
}

Schema 2 renamed id 3 and widened it from integer to bigint. It also dropped id 4 and added id 5, and not one data file was rewritten. A reader on schema 2 handles a file written under schema 1 like this:

Table L is a directory of files whose engine matches columns by name; table T is a table format with field ids. Both hold 3 years of orders. You rename amt to amount in each. What does SUM(amount) return?

• T returns the full total; L sums only rows written after the rename, because older files have no amount
• Both return the full total, because a rename changes nothing stored in the data files
• Both sum only rows written after the rename until old files are rewritten with the new name
• L returns the full total, while T returns NULL for old rows until their files are rewritten

Which type changes stay metadata-only

A widening is safe when every value the old type can hold is exactly representable in the new one, so readers upcast while decoding. Any change that could lose or reinterpret a value needs new files: add a new column, fill it by rewriting, move readers over, then drop the old one.

Metadata-only

• integer to bigint
• real to double
• decimal(10,2) to decimal(12,2)
• required column to optional

Not metadata-only

• bigint to integer
• varchar to integer
• decimal(10,2) to decimal(10,4)
• optional column to required

Which change to orders cannot commit as a metadata-only schema update?

• Adding a nullable promo_code column that older files will read as NULL
• Renaming currency to currency_code while dashboards still query the old name
• Narrowing amount from bigint to integer when some stored values exceed the integer range
• Widening a quantity column from integer to bigint so it can hold larger counts

The drop and re-add trap

Ids also make a mistake expensive. Drop legacy_flag (id 4), regret it a week later and add legacy_flag boolean again: the new column gets id 6. Old files hold their values under id 4, readers ask for id 6, and every historical row reads NULL. The bytes still sit in the old files, but no schema maps a column to them, so recovery means a rewrite that copies id 4 values into id 6.

About this lesson

This section is part of the Storage Layers: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.