Schema Drift: Intermediate
What the registry stores
Schemas are grouped by subject, usually 1 per topic, such as trip_completed-value. Each distinct schema receives a schema ID that is unique across the whole registry. A subject holds an ordered list of versions, and each version points to one of those IDs, so an identical schema registered under 2 subjects shares 1 ID. The subject also carries a compatibility mode: the rule every new version must pass before the registry accepts it.
Deploy time and runtime
Before a deploy, the check asks the registry whether the proposed schema is compatible. At runtime, trip_service stamps each message with a schema ID, and each consumer fetches the matching schema from the registry.
The 4 compatibility modes
| Mode | Guarantee | Passes | Fails |
|---|---|---|---|
| BACKWARD | Readers on the new schema decode old messages | Delete a field; add a field with a default | Add a field with no default |
| FORWARD | Readers on the old schema decode new messages | Add a field; delete a field that has a default | Delete a field with no default |
| FULL | Both directions hold | Add or delete fields that have defaults | Add or delete a field with no default |
| NONE | No guarantee | Any change | Nothing |
The trip_completed-value subject uses FORWARD compatibility. A proposed version 3 deletes driver_id, a required string with no default in version 2. What does the check return?
The mode also fixes who deploys first. Under BACKWARD, consumers upgrade first, since a new reader can decode old messages. Under FORWARD, the producer upgrades first, since old readers can decode new messages. FULL lets either side go first, which suits a topic read by several teams that ship on separate schedules.
Each mode compares a candidate with the latest version only. The transitive variants, such as FULL_TRANSITIVE, compare it with every registered version. Choose a transitive mode when consumers replay history: a topic that retains 30 days can hold messages written under 4 versions, and a replay from day 1 decodes all of them.
- ▸Renaming
faretofare_centsdeletes one field and adds another. - ▸Under
FULL, the addedfare_centshas no default, so new readers cannot fill it in from old messages. - ▸The deleted
farehas no default either, so old readers find nothing to read in new messages. - ▸The check fails on both counts. The next section makes the change anyway, in compatible steps.
Which proposed version 3 change passes under FULL compatibility?
The registry blocks the rename, and the business still needs it. Today fare is a string such as 12.50 USD, and each of the topic's 3 consumers parses it with code of its own. The target is a long fare_cents plus a string fare_currency. Expand-contract makes that change in 4 deploys, each compatible with the one before it, so no team waits on another team's release.
- The producer and 3 consumers ship inside one release window
- A rollback has to undo all 4 deploys together
- A consumer that misses the window breaks on the first new message
- Messages already on the topic still carry the old shape
- Each team ships on its own schedule
- Each phase rolls back on its own
- Old and new fields coexist for as long as readers need
- The old field leaves only after its last reader has moved
The 4 phases
| Phase | Producer writes | Consumers read | Schema |
|---|---|---|---|
| 1. Expand | `fare`, `fare_cents`, `fare_currency` | `fare` | v3: adds 2 fields with null defaults, gives `fare` a default |
| 2. Migrate | all 3 fields | `fare`, while history is backfilled | v3, unchanged |
| 3. Switch | all 3 fields | each consumer moves to `fare_cents` on its own date | v3, unchanged |
| 4. Contract | `fare_cents`, `fare_currency` | `fare_cents` | v4: deletes `fare` |
Expand: add beside the old field
Version 3 adds fare_cents and fare_currency, both nullable with a default of null, and gives the existing fare a default of an empty string. Every change carries a default, so FULL passes. The producer now writes all 3 fields on every message. Nobody reads the new ones yet, so rolling back to version 2 breaks no reader. The default on fare does nothing today; it is what lets phase 4 delete the field.
Migrate: give history the new fields
The curated trips table holds 2 years of rows that only have fare. A backfill computes fare_cents and fare_currency for each trip_date partition and overwrites that partition, so a retried day produces the same rows. Before any consumer depends on the backfilled values, compare the backfill's parsing with what the producer wrote itself on recent days, where both columns exist.
During phase 2, the backfill for partition 2025-03-14 fails after writing half its rows, and the orchestrator retries the task. What makes the retry safe?
Switch: consumers move one by one
Each consumer team changes its reads to fare_cents, ships, and watches its own outputs. The earnings rollup might switch in week 1 and the fraud model in week 3, with the finance export last. Keep a written list of the topic's 3 consumers with a sign-off column: the registry records schemas and keeps no record of who reads which field.
Contract: remove what nobody reads
When the last sign-off lands, the producer ships version 4, which stops writing fare and deletes it from the schema. Because fare gained a default in phase 1, FULL accepts the deletion. Dropping the fare column from trips comes last, once a week passes with no query reading it.
fare, so rolling the producer back restores the field for new messages only. Wait out a full release cycle of the slowest consumer before taking that step.Two of the 3 consumers of trip_completed read fare_cents. The third, the monthly finance export, still parses fare. The producer ships version 4, which stops writing fare. What does the finance export see on its next run?
Every trip event carries 2 timestamps that answer different questions. Event time is when the thing happened, stamped by the phone: the trip ended at 08:47. Processing time is when the pipeline handled the event: the streaming job saw it at 09:12. The gap between them is skew. On a good network skew is a few seconds; for a phone in a garage or on a flight it runs to hours.
| trip | ended_at (event time) | processed_at (processing time) | skew |
|---|---|---|---|
| T1 | 08:58:00 | 08:58:03 | 3s |
| T2 | 08:59:40 | 09:00:05 | 25s |
| T3 | 08:47:00 | 09:12:00 | 25m |
| T4 | 09:10:00 | 09:10:02 | 2s |
| T5 | 09:20:00 | 09:20:01 | 1s |
| T6 | 08:55:00 | 10:02:00 | 67m |
| hour | trips |
|---|---|
| 08:00 | 4 |
| 09:00 | 2 |
| hour | trips |
|---|---|
| 08:00 | 1 |
| 09:00 | 4 |
| 10:00 | 1 |
The 2 queries differ only in the column inside date_trunc. This one buckets by event time; putting processed_at in place of ended_at produces the right-hand table.
The partition column inherits the same choice, and the query above assumes trip_date comes from ended_at. If trip_date comes from processed_at, a trip that ends at 23:50 and uploads at 00:40 lands in the next day's partition, and an event-time query for one day must scan 2 partitions to find all its trips.
Which clock answers which question
Ask whether the question is about the world or about the pipeline. Questions about what riders and drivers did use event time. Questions about load, delay and cost inside the system use processing time.
| Consumer question | Clock | Reason |
|---|---|---|
| Driver earnings for 08:00 to 09:00 | event time | Pay follows when the trip happened |
| Events ingested per minute, for capacity planning | processing time | Load lands when events arrive |
| Fraud rule: 5 trips on one card within 10 minutes | event time | The rule describes rider behaviour |
| Delay from phone to table, p95 | both | Delay is processed_at minus ended_at |
| Alert when the job falls behind | processing time | Falling behind is a property of the job |
This query feeds the driver earnings page, which shows pay per hour of driving. What is wrong with it?
- ▸It is only as accurate as the clock that stamped it, and phone clocks drift or get set by hand.
- ▸A producer that stamps at send time records upload time under the name of event time.
- ▸Quarantine event times far in the future: they point to a bad clock.
- ▸Keep
processed_aton every row so skew stays measurable after the fact.
During a concert surge, the on-call engineer wants a chart showing whether the streaming job keeps up with incoming events. Which timestamp should the chart bucket by?
A watermark is a timestamp in event time that the engine carries alongside the stream. When it reads W, the engine is asserting: no further events with event time earlier than W will arrive. A window covering [09:00, 09:05) counts as complete once the watermark reaches 09:05, and at that moment the engine emits, or fires, the window's total.
Max event time seen, minus a lag
| arrives | event | event_time | amount | watermark after | what happens |
|---|---|---|---|---|---|
| 09:02:18 | E1 | 09:02:14 | 100 | 09:01:14 | joins [09:00, 09:05), total 100 |
| 09:04:30 | E2 | 09:03:55 | 200 | 09:02:55 | joins [09:00, 09:05), total 300 |
| 09:06:02 | E5 | 09:06:01 | 40 | 09:05:01 | opens [09:05, 09:10); [09:00, 09:05) fires with 300 |
| 09:34:00 | E3 | 09:02:14 | 50 | 09:05:01 | event time sits behind the watermark: late |
Because this watermark is built from event times, a quiet stream holds it still. Window [09:05, 09:10) fires only when an event stamped at or after 09:11:00 arrives, however much wall-clock time passes first.
A job sums 5-minute windows with a 60-second watermark lag. Events arrive in this order, stamped 10:03:10, 10:05:30, 10:06:20, then 10:04:50. What happens to window [10:00, 10:05)?
Choosing the lag
- Totals publish about 10 seconds after a window ends
- Any event delayed past 10 seconds counts as late
- Open windows sit in memory briefly
- Suits alerting, where a fast rough answer beats a slow exact one
- Totals publish about 30 minutes after a window ends
- Only events delayed past 30 minutes count as late
- Every key holds 30 minutes of open windows in memory
- Suits billing, where completeness outranks speed
The lag trades latency for completeness, so set it from measured skew. If the p99 of processed_at minus ended_at is 40 seconds, a 60-second lag leaves fewer than 1 event in 100 late in a steady stream, and every total waits about a minute past its window's end.
A team raises the watermark lag on its revenue job from 60 seconds to 10 minutes. What should it expect?
Allowed lateness is the second setting on a window. After the watermark fires a window, the engine keeps that window's state for a further stretch of event time, 60 minutes in this job. A late event that lands inside the stretch updates the total, and the window fires again with the corrected figure. An event that turns up after the stretch finds no state left to update and goes to late_events, a dead letter table where a person can find it.
| Class | Arrives while the watermark is | Example | Result for [09:00, 09:05) |
|---|---|---|---|
| On-time | before 09:05 | E1, E2 | In the first firing: 300 |
| Slightly late | from 09:05 up to 10:05 | E3, at 09:34 | Total becomes 350, window fires again |
| Very late | at or past 10:05 | E4, at 11:15 | State already discarded; E4 goes to late_events |
The animation replays all 6 events through the revenue_5m job in arrival order. Run the 60-minute variant, then the variant with no allowed lateness, and watch where E3 ends up each time.
Lateness runs on the watermark
E6 decided E4's fate. Stamped 10:06:30, E6 moved the watermark to 10:05:30, past the window's end plus 60 minutes, so the engine discarded the window's state and E4 found nothing to join at 11:15. Had E6 never arrived, the watermark would still read 09:05:01 and E4 would have been accepted, however late the wall clock said it was. With allowed lateness at 0, the state goes the moment the window fires, so E3 and E4 both land in late_events and the total stays at 300.
The window logic in plain Python
| window | total | fires |
|---|---|---|
| [09:00, 09:05) | 350 | 2 |
| [09:05, 09:10) | 40 | 1 |
The function returns late as ['E4']. Window [10:05, 10:10) holds E6's 60 and has not fired, because the watermark stands at 10:05:30.
Every re-fire replaces the last one
| window | total |
|---|---|
| 09:00-09:05 | 300 |
| 09:00-09:05 | 350 |
| window | total | fires |
|---|---|---|
| 09:00-09:05 | 350 | 2 |
A team's sink appended every firing to revenue_firings, one row per firing with a fired_at timestamp. Late refunds can lower a total on re-fire. Which aggregation returns the current total per window?
What an hour of lateness costs
- ▸Route them to
late_events; a silent drop hides how often lateness runs out. - ▸Chart the daily count of
late_eventsnext to the window totals. - ▸Let the batch rerun of the last 7 days fold them into the curated
tripstable. - ▸If
late_eventskeeps growing, measure skew again before raising allowed lateness.
A job uses 5-minute windows, a 60-second lag and 60 minutes of allowed lateness. The watermark reads 14:40:00. Three events then arrive, stamped 14:03:00, then 15:06:00, then 14:04:00. What happens to window [14:00, 14:05)?
> Picture Uber's earnings pipeline during the fare rename. Version 3 of trip_completed passes the registry's FULL check because every change carries a default. While the 3 consumers switch over the following month, the streaming earnings job keeps summing 5-minute windows by ended_at, with a 60-second watermark lag and 60 minutes of allowed lateness, and upserts each firing into the earnings table. Trips uploaded after their window's lateness runs out land in late_events, and the daily rerun of the last 7 days folds them into trips.
ended_at, because driver pay follows when a trip happened.processed_at and ended_at.late_events like a dead letter queue, and the 7-day batch rerun repairs the curated table.Registries, expand-contract and watermarks: how pipelines survive renamed fields and fares that arrive an hour late.
- Category
- Pipeline Architecture
- Difficulty
- intermediate
- Duration
- 32 minutes
- Challenges
- 12 hands-on challenges
Topics covered: Schema Registries: Where They Live, The Expand-Contract Pattern, Event Time Versus Processing Time, Watermarks: The Engine's Promise, 1-Hour Allowed Lateness
Lesson Sections
- Schema Registries: Where They Live (concepts: Schema Evolution)
A schema registry is a small service that stores every version of every event schema and checks each proposed version against the versions before it. In the beginner lesson, a producer changed its event shape and a consumer broke at runtime. A registry moves that failure to deploy time, where it costs a red build instead of a broken dashboard. What the registry stores The registry also changes what travels on the topic. Each message carries a short schema ID ahead of its payload, in place of the
- The Expand-Contract Pattern (concepts: Schema Evolution)
The 4 phases Expand: add beside the old field Migrate: give history the new fields Switch: consumers move one by one Contract: remove what nobody reads
- Event Time Versus Processing Time (concepts: Stream Processing)
Count those 6 trips per hour and the 2 clocks disagree about every hour. T3 and T6 both ended in the 08:00 hour, yet T3 reached the pipeline at 09:12 and T6 at 10:02. Which clock answers which question
- Watermarks: The Engine's Promise (concepts: Late-Arriving Data)
A daily batch job knows yesterday is finished when the last file lands. A streaming job that sums revenue in 5-minute event-time windows gets no such signal, and T6 showed that a trip can surface 67 minutes after it ended. Before the job publishes a total for 09:00 to 09:05, it needs a rule for deciding it has waited long enough. A watermark is that rule. Max event time seen, minus a lag Most engines compute a heuristic watermark from the data itself: the largest event time seen so far, minus a
- 1-Hour Allowed Lateness (concepts: Late-Arriving Data)
Lateness runs on the watermark The window logic in plain Python The function below holds the whole policy: a 5-minute window, a 60-second lag and 60 minutes of allowed lateness. For each event it checks whether the window's lateness has run out, updates the total, re-fires a window that already fired, then advances the watermark and fires or discards windows. Every re-fire replaces the last one Each firing emits the window's full current total, and a later firing supersedes an earlier one. A sin