What About Schema Changes?
Concepts covered: Schema Evolution
Your incremental pipeline is running smoothly. Then on Tuesday at 2 AM, the backend team adds a new column to the source table. Your pipeline either crashes, silently drops the new column, or - if you are lucky - handles it gracefully. The interviewer who asks "what about schema changes?" is testing whether you have experienced this pain. Your answer should prove that you have.
The Schema Change Taxonomy
| Change Type | Impact | Typical Handling |
|---|---|---|
| Add column | Low | Append to target, backfill NULLs |
| Rename column | Medium | Alias in transformation layer |
| Change data type | High | Migration script + backfill |
| Drop column | High | Deprecation period, then remove |
| Change primary key | Critical | Full rebuild required |
Not all schema changes are equal. The interviewer wants to hear you classify them by impact level, not treat them all the same. Adding a column is almost always safe. Dropping or renaming a column can break every downstream consumer. Senior signal: mention that changing a primary key requires a full rebuild, and that most teams underestimate this.
Schema Registries
A schema registry is a versioned catalog of every schema your pipeline has seen. When a source schema changes, the registry detects it, classifies the change as breaking or non-breaking, and either auto-applies it or blocks the pipeline and alerts the team. Schema registries like Confluent Schema Registry (for Kafka/Avro) are the tools to name-drop. The interviewer wants to hear that you have a system, not just hope.
Backwards compatibility means new data can be read by old consumers. A new nullable column is backwards compatible - old consumers simply ignore it. A removed column is not - old consumers that expect it will fail. The follow-up question: "How do you handle a breaking change?" Your answer: deprecation period, consumer notification, and a migration window. Never a surprise ALTER TABLE.
- Do
- Avoid
Consumer Notification
The hardest part of schema evolution is not technical - it is organizational. When you add a column to an upstream table, who needs to know? Every pipeline, dashboard, and ML model that reads from any downstream table that might propagate the change. Schema registries automate detection, but notification requires a human process: a changelog, a Slack channel, a review step in CI. Say this and you are demonstrating real production experience.
About this lesson
This section is part of the Incremental Loading: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.