When a Producer Adds a Column

Concepts covered: Schema Evolution

The field your loader never expected

A producer is the system that writes the data a pipeline reads. Here it is the trip service, which emits an event per completed trip, and the loader is the step that copies those events into a warehouse table called trips. The loader was built when every event carried 4 fields, trip_id and city plus fare_usd and completed_at. Those names and the type of value each holds are the schema the loader expects.

{
  "trip_id": "t_88213",
  "city": "Lisbon",
  "fare_usd": 14.20,
  "completed_at": "2026-03-02T23:50:00Z",
  "tip_usd": 2.00
}

Then the producer team ships tipping, and each event gains tip_usd. Their change is correct: the app needs the field and their tests pass. Nobody on that team knows your loader exists, and nothing in the event announces a new shape. This is schema drift: the structure of data changes over time, and each system reading it finds out when the new shape arrives.

3 reactions to an unknown field

When the loader meets a field its table has no column for, it does 1 of 3 things, whether its author chose on purpose or inherited a library default. Learn which one yours does before the next producer release.

def load_event(event, table_columns, mode):
    unknown = [name for name in event if name not in table_columns]
    if unknown and mode == "reject":
        raise ValueError(f"unknown fields: {unknown}")
    if unknown and mode == "evolve":
        for name in unknown:
            add_column(name, infer_type(event[name]))
            table_columns.append(name)
    row = {name: event.get(name) for name in table_columns}
    insert_row(row)

What each reaction costs

Here is trips after a batch of 3 events: an old event with no tip, then 2 new events tipping 2.00 and 0.00.

ReactionNew rows in tripstip_usd columnWho notices, and when
Reject0 of 3: the batch failsAbsentOn-call, at the failed run
Ignore3 of 3Absent, values lostAn analyst, weeks later, asking for tips
Evolve3 of 3null, 2.00, 0.00Whoever reviews new columns, if anyone does

A nightly loader keeps succeeding after a producer release, with row counts that match the source. A week later an analyst finds the new promo_code field nowhere in the warehouse. Which reaction is the loader using?

No reaction is right everywhere. A table feeding driver payouts should reject, because a stale number is safer than a silently incomplete one. A clickstream table with hundreds of loosely owned fields often ignores, with an alert naming each dropped field, because stopping it would page someone every week. A table where analysts want new fields fast can evolve, if someone reviews each new column.

Your team owns payouts, which decides what drivers are paid each week, and app_clicks, which collects UI events from 40 product teams that add fields often. Which pairing of reactions fits best?

About this lesson

This section is part of the Schema Drift: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.