When a Producer Adds a Column
Concepts covered: Schema Evolution
The field your loader never expected
A producer is the system that writes the data a pipeline reads. Here it is the trip service, which emits an event per completed trip, and the loader is the step that copies those events into a warehouse table called trips. The loader was built when every event carried 4 fields, trip_id and city plus fare_usd and completed_at. Those names and the type of value each holds are the schema the loader expects.
{
"trip_id": "t_88213",
"city": "Lisbon",
"fare_usd": 14.20,
"completed_at": "2026-03-02T23:50:00Z",
"tip_usd": 2.00
}Then the producer team ships tipping, and each event gains tip_usd. Their change is correct: the app needs the field and their tests pass. Nobody on that team knows your loader exists, and nothing in the event announces a new shape. This is schema drift: the structure of data changes over time, and each system reading it finds out when the new shape arrives.
3 reactions to an unknown field
When the loader meets a field its table has no column for, it does 1 of 3 things, whether its author chose on purpose or inherited a library default. Learn which one yours does before the next producer release.
def load_event(event, table_columns, mode):
unknown = [name for name in event if name not in table_columns]
if unknown and mode == "reject":
raise ValueError(f"unknown fields: {unknown}")
if unknown and mode == "evolve":
for name in unknown:
add_column(name, infer_type(event[name]))
table_columns.append(name)
row = {name: event.get(name) for name in table_columns}
insert_row(row)What each reaction costs
- Reject: No wrong data reaches the table, but every dashboard reading trips goes stale. Retrying changes nothing: this is a permanent failure.
- Ignore: The run succeeds and trips stays fresh, while every tip value is thrown away. Without a raw copy of the events, those values are gone.
- Evolve: Tips land from the first event. The risk is unreviewed growth: a typo such as tipUsd in a single app version becomes a second permanent column.
Here is trips after a batch of 3 events: an old event with no tip, then 2 new events tipping 2.00 and 0.00.
| Reaction | New rows in trips | tip_usd column | Who notices, and when |
|---|---|---|---|
| Reject | 0 of 3: the batch fails | Absent | On-call, at the failed run |
| Ignore | 3 of 3 | Absent, values lost | An analyst, weeks later, asking for tips |
| Evolve | 3 of 3 | null, 2.00, 0.00 | Whoever reviews new columns, if anyone does |
A nightly loader keeps succeeding after a producer release, with row counts that match the source. A week later an analyst finds the new promo_code field nowhere in the warehouse. Which reaction is the loader using?
- Reject: the loader failed on the unknown field, and someone reran it without the field.
- Ignore: it loaded the fields its table knows and dropped promo_code without failing.
- Evolve: it added a promo_code column, but the column stays empty for new rows.
- Reject: the warehouse refused the new column, so the loader quietly kept the old schema.
No reaction is right everywhere. A table feeding driver payouts should reject, because a stale number is safer than a silently incomplete one. A clickstream table with hundreds of loosely owned fields often ignores, with an alert naming each dropped field, because stopping it would page someone every week. A table where analysts want new fields fast can evolve, if someone reviews each new column.
Your team owns payouts, which decides what drivers are paid each week, and app_clicks, which collects UI events from 40 product teams that add fields often. Which pairing of reactions fits best?
- Evolve for both, so no field from any producer is ever lost.
- Reject for both, so no table ever holds data nobody reviewed.
- Ignore for payouts, reject for app_clicks, so money keeps flowing.
- Reject for payouts, ignore with an alert for app_clicks.
About this lesson
This section is part of the Schema Drift: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.