IntermediatePipeline Architecture · 32 min · 12 challenges

Schema Drift: Intermediate

At Uber, a driver's phone climbs out of an underground garage at 09:34 and uploads a fare from a trip that ended at 09:02. The earnings pipeline published its 09:00 to 09:05 revenue total 28 minutes earlier. Reopen that total, drop the fare, or park it? The same app release also renamed the fare field. Two kinds of promise settle both: one about schema, one about time.

Schema Registries: Where They Live

Daily Life
Interviews
A schema registry is a small service that stores every version of every event schema and checks each proposed version against the versions before it. In the beginner lesson, a producer changed its event shape and a consumer broke at runtime. A registry moves that failure to deploy time, where it costs a red build instead of a broken dashboard.

What the registry stores

Schemas are grouped by subject, usually 1 per topic, such as trip_completed-value. Each distinct schema receives a schema ID that is unique across the whole registry. A subject holds an ordered list of versions, and each version points to one of those IDs, so an identical schema registered under 2 subjects shares 1 ID. The subject also carries a compatibility mode: the rule every new version must pass before the registry accepts it.

{
"subject": "trip_completed-value",
"compatibility": "FULL",
"versions": [
{
"version": 1,
"schema_id": 412,
"fields": [
{
"name": "trip_id",
"type": "string"
},
{
"name": "driver_id",
"type": "string"
},
{
"name": "fare",
"type": "string"
},
{
"name": "ended_at",
"type": "timestamp"
}
]
},
{
"version": 2,
"schema_id": 587,
"fields": [
{
"name": "trip_id",
"type": "string"
},
{
"name": "driver_id",
"type": "string"
},
{
"name": "fare",
"type": "string"
},
{
"name": "ended_at",
"type": "timestamp"
},
{
"name": "tip_cents",
"type": "long",
"default": 0
}
]
}
]
}
The registry also changes what travels on the topic. Each message carries a short schema ID ahead of its payload, in place of the full schema. A consumer that meets ID 587 for the first time fetches that schema from the registry once, caches it, and decodes every later message stamped 587 with it.

Deploy time and runtime

producer app
trip service
pre-deploy check
compat check
ERRORfail the deploy
event topic
trip completed
registry service
schema registry
streaming job
earnings rollup
streaming job
fraud scoring
monthly batch
finance export

Before a deploy, the check asks the registry whether the proposed schema is compatible. At runtime, trip_service stamps each message with a schema ID, and each consumer fetches the matching schema from the registry.

The check that matters runs before the deploy. When trip_service is about to ship a new event shape, a build step submits the proposed schema to the registry and asks whether it passes the subject's compatibility mode. A failure stops the release, so the incompatible shape never produces a message. Runtime is the backstop: a producer that skips the check and has auto-registration turned on tries to register the new version when it first serializes a message, and that send fails with a compatibility error where it expected a schema ID.

The 4 compatibility modes

ModeGuaranteePassesFails
BACKWARDReaders on the new schema decode old messagesDelete a field; add a field with a defaultAdd a field with no default
FORWARDReaders on the old schema decode new messagesAdd a field; delete a field that has a defaultDelete a field with no default
FULLBoth directions holdAdd or delete fields that have defaultsAdd or delete a field with no default
NONENo guaranteeAny changeNothing
Multiple Choice

The trip_completed-value subject uses FORWARD compatibility. A proposed version 3 deletes driver_id, a required string with no default in version 2. What does the check return?

The mode also fixes who deploys first. Under BACKWARD, consumers upgrade first, since a new reader can decode old messages. Under FORWARD, the producer upgrades first, since old readers can decode new messages. FULL lets either side go first, which suits a topic read by several teams that ship on separate schedules.

Each mode compares a candidate with the latest version only. The transitive variants, such as FULL_TRANSITIVE, compare it with every registered version. Choose a transitive mode when consumers replay history: a topic that retains 30 days can hold messages written under 4 versions, and a replay from day 1 decodes all of them.

A rename, as the registry sees it
  • ▸Renaming fare to fare_cents deletes one field and adds another.
  • ▸Under FULL, the added fare_cents has no default, so new readers cannot fill it in from old messages.
  • ▸The deleted fare has no default either, so old readers find nothing to read in new messages.
  • ▸The check fails on both counts. The next section makes the change anyway, in compatible steps.
Multiple Choice

Which proposed version 3 change passes under FULL compatibility?

The Expand-Contract Pattern

Daily Life
Interviews

The registry blocks the rename, and the business still needs it. Today fare is a string such as 12.50 USD, and each of the topic's 3 consumers parses it with code of its own. The target is a long fare_cents plus a string fare_currency. Expand-contract makes that change in 4 deploys, each compatible with the one before it, so no team waits on another team's release.

✓One synchronized deploy
  • The producer and 3 consumers ship inside one release window
  • A rollback has to undo all 4 deploys together
  • A consumer that misses the window breaks on the first new message
  • Messages already on the topic still carry the old shape
✗Expand-contract
  • Each team ships on its own schedule
  • Each phase rolls back on its own
  • Old and new fields coexist for as long as readers need
  • The old field leaves only after its last reader has moved

The 4 phases

PhaseProducer writesConsumers readSchema
1. Expand`fare`, `fare_cents`, `fare_currency``fare`v3: adds 2 fields with null defaults, gives `fare` a default
2. Migrateall 3 fields`fare`, while history is backfilledv3, unchanged
3. Switchall 3 fieldseach consumer moves to `fare_cents` on its own datev3, unchanged
4. Contract`fare_cents`, `fare_currency``fare_cents`v4: deletes `fare`
//

Expand: add beside the old field

Version 3 adds fare_cents and fare_currency, both nullable with a default of null, and gives the existing fare a default of an empty string. Every change carries a default, so FULL passes. The producer now writes all 3 fields on every message. Nobody reads the new ones yet, so rolling back to version 2 breaks no reader. The default on fare does nothing today; it is what lets phase 4 delete the field.

//

Migrate: give history the new fields

The curated trips table holds 2 years of rows that only have fare. A backfill computes fare_cents and fare_currency for each trip_date partition and overwrites that partition, so a retried day produces the same rows. Before any consumer depends on the backfilled values, compare the backfill's parsing with what the producer wrote itself on recent days, where both columns exist.

SELECT
trip_date,
COUNT(*) AS rows_checked,
COUNT_IF(
fare_cents <> CAST(
ROUND(CAST(SPLIT_PART(fare, ' ', 1) AS DOUBLE) * 100)
AS BIGINT
)
) AS mismatched_rows
FROM trips
WHERE trip_date >= DATE '2026-10-01'
GROUP BY trip_date
ORDER BY trip_date
Multiple Choice

During phase 2, the backfill for partition 2025-03-14 fails after writing half its rows, and the orchestrator retries the task. What makes the retry safe?

//

Switch: consumers move one by one

Each consumer team changes its reads to fare_cents, ships, and watches its own outputs. The earnings rollup might switch in week 1 and the fraud model in week 3, with the finance export last. Keep a written list of the topic's 3 consumers with a sign-off column: the registry records schemas and keeps no record of who reads which field.

//

Contract: remove what nobody reads

When the last sign-off lands, the producer ships version 4, which stops writing fare and deletes it from the schema. Because fare gained a default in phase 1, FULL accepts the deletion. Dropping the fare column from trips comes last, once a week passes with no query reading it.

TIP
Contract is the one phase with a one-way door. Messages written under version 4 have no fare, so rolling the producer back restores the field for new messages only. Wait out a full release cycle of the slowest consumer before taking that step.
Multiple Choice

Two of the 3 consumers of trip_completed read fare_cents. The third, the monthly finance export, still parses fare. The producer ships version 4, which stops writing fare. What does the finance export see on its next run?

Event Time Versus Processing Time

Daily Life
Interviews

Every trip event carries 2 timestamps that answer different questions. Event time is when the thing happened, stamped by the phone: the trip ended at 08:47. Processing time is when the pipeline handled the event: the streaming job saw it at 09:12. The gap between them is skew. On a good network skew is a few seconds; for a phone in a garage or on a flight it runs to hours.

tripended_at (event time)processed_at (processing time)skew
T108:58:0008:58:033s
T208:59:4009:00:0525s
T308:47:0009:12:0025m
T409:10:0009:10:022s
T509:20:0009:20:011s
T608:55:0010:02:0067m
Count those 6 trips per hour and the 2 clocks disagree about every hour. T3 and T6 both ended in the 08:00 hour, yet T3 reached the pipeline at 09:12 and T6 at 10:02.
By event hour
hourtrips
08:004
09:002
By processing hour
hourtrips
08:001
09:004
10:001

The 2 queries differ only in the column inside date_trunc. This one buckets by event time; putting processed_at in place of ended_at produces the right-hand table.

SELECT
DATE_TRUNC('hour', ended_at) AS event_hour,
COUNT(*) AS trips
FROM trips
WHERE trip_date = DATE '2026-10-24'
GROUP BY DATE_TRUNC('hour', ended_at)
ORDER BY event_hour

The partition column inherits the same choice, and the query above assumes trip_date comes from ended_at. If trip_date comes from processed_at, a trip that ends at 23:50 and uploads at 00:40 lands in the next day's partition, and an event-time query for one day must scan 2 partitions to find all its trips.

Which clock answers which question

Ask whether the question is about the world or about the pipeline. Questions about what riders and drivers did use event time. Questions about load, delay and cost inside the system use processing time.

Consumer questionClockReason
Driver earnings for 08:00 to 09:00event timePay follows when the trip happened
Events ingested per minute, for capacity planningprocessing timeLoad lands when events arrive
Fraud rule: 5 trips on one card within 10 minutesevent timeThe rule describes rider behaviour
Delay from phone to table, p95bothDelay is processed_at minus ended_at
Alert when the job falls behindprocessing timeFalling behind is a property of the job
Multiple Choice

This query feeds the driver earnings page, which shows pay per hour of driving. What is wrong with it?

SELECT
driver_id,
DATE_TRUNC('hour', processed_at) AS pay_hour,
SUM(fare_cents) AS earned_cents
FROM trips
WHERE trip_date = DATE '2026-10-24'
GROUP BY driver_id, DATE_TRUNC('hour', processed_at)
Event time can be wrong too
  • ▸It is only as accurate as the clock that stamped it, and phone clocks drift or get set by hand.
  • ▸A producer that stamps at send time records upload time under the name of event time.
  • ▸Quarantine event times far in the future: they point to a bad clock.
  • ▸Keep processed_at on every row so skew stays measurable after the fact.
Multiple Choice

During a concert surge, the on-call engineer wants a chart showing whether the streaming job keeps up with incoming events. Which timestamp should the chart bucket by?

Watermarks: The Engine's Promise

Daily Life
Interviews
A daily batch job knows yesterday is finished when the last file lands. A streaming job that sums revenue in 5-minute event-time windows gets no such signal, and T6 showed that a trip can surface 67 minutes after it ended. Before the job publishes a total for 09:00 to 09:05, it needs a rule for deciding it has waited long enough. A watermark is that rule.

A watermark is a timestamp in event time that the engine carries alongside the stream. When it reads W, the engine is asserting: no further events with event time earlier than W will arrive. A window covering [09:00, 09:05) counts as complete once the watermark reaches 09:05, and at that moment the engine emits, or fires, the window's total.

Max event time seen, minus a lag

Most engines compute a heuristic watermark from the data itself: the largest event time seen so far, minus a fixed lag that absorbs ordinary out-of-order delivery. With a 60-second lag, an event stamped 09:06:01 moves the watermark to 09:05:01. It never moves backward, and it moves forward only when an event newer than every earlier one arrives.
arriveseventevent_timeamountwatermark afterwhat happens
09:02:18E109:02:1410009:01:14joins [09:00, 09:05), total 100
09:04:30E209:03:5520009:02:55joins [09:00, 09:05), total 300
09:06:02E509:06:014009:05:01opens [09:05, 09:10); [09:00, 09:05) fires with 300
09:34:00E309:02:145009:05:01event time sits behind the watermark: late
Read the last row closely. E3 ended at 09:02:14, before the watermark of 09:05:01, so the engine's assertion turned out false. A heuristic watermark is a bet about the network, and a phone in a garage loses that bet. The watermark held at 09:05:01 as E3 arrived, since E3 is older than events the engine had already seen. What the engine does with E3 is a separate setting, the subject of the next section.

Because this watermark is built from event times, a quiet stream holds it still. Window [09:05, 09:10) fires only when an event stamped at or after 09:11:00 arrives, however much wall-clock time passes first.

Multiple Choice

A job sums 5-minute windows with a 60-second watermark lag. Events arrive in this order, stamped 10:03:10, 10:05:30, 10:06:20, then 10:04:50. What happens to window [10:00, 10:05)?

Choosing the lag

✓10-second lag
  • Totals publish about 10 seconds after a window ends
  • Any event delayed past 10 seconds counts as late
  • Open windows sit in memory briefly
  • Suits alerting, where a fast rough answer beats a slow exact one
✗30-minute lag
  • Totals publish about 30 minutes after a window ends
  • Only events delayed past 30 minutes count as late
  • Every key holds 30 minutes of open windows in memory
  • Suits billing, where completeness outranks speed

The lag trades latency for completeness, so set it from measured skew. If the p99 of processed_at minus ended_at is 40 seconds, a 60-second lag leaves fewer than 1 event in 100 late in a steady stream, and every total waits about a minute past its window's end.

Multiple Choice

A team raises the watermark lag on its revenue job from 60 seconds to 10 minutes. What should it expect?

1-Hour Allowed Lateness

Daily Life
Interviews

Allowed lateness is the second setting on a window. After the watermark fires a window, the engine keeps that window's state for a further stretch of event time, 60 minutes in this job. A late event that lands inside the stretch updates the total, and the window fires again with the corrected figure. An event that turns up after the stretch finds no state left to update and goes to late_events, a dead letter table where a person can find it.

ClassArrives while the watermark isExampleResult for [09:00, 09:05)
On-timebefore 09:05E1, E2In the first firing: 300
Slightly latefrom 09:05 up to 10:05E3, at 09:34Total becomes 350, window fires again
Very lateat or past 10:05E4, at 11:15State already discarded; E4 goes to late_events

The animation replays all 6 events through the revenue_5m job in arrival order. Run the 60-minute variant, then the variant with no allowed lateness, and watch where E3 ends up each time.

A window that has fired still takes late events for 60 minutes09:02:00
Kafka
mobile_events
Beam
revenue_5m
LATENESS1 h
BigQuery
revenue_by_window
dead letter
late_events
records0 records
#
id
event_time
amount
event-time windows5 min
09:0009:05…10:0510:10
09:00–09:05
total
09:05–09:10
total
10:05–10:10
total
no watermark yet
revenue_by_window0 rows
window
total
fires
records0 records
#
id
event_time
amount
One window, three kinds of late
1/22

Lateness runs on the watermark

E6 decided E4's fate. Stamped 10:06:30, E6 moved the watermark to 10:05:30, past the window's end plus 60 minutes, so the engine discarded the window's state and E4 found nothing to join at 11:15. Had E6 never arrived, the watermark would still read 09:05:01 and E4 would have been accepted, however late the wall clock said it was. With allowed lateness at 0, the state goes the moment the window fires, so E3 and E4 both land in late_events and the total stays at 300.

The window logic in plain Python

The function below holds the whole policy: a 5-minute window, a 60-second lag and 60 minutes of allowed lateness. For each event it checks whether the window's lateness has run out, updates the total, re-fires a window that already fired, then advances the watermark and fires or discards windows.
from datetime import datetime, timedelta
WINDOW = timedelta(minutes=5)
WATERMARK_LAG = timedelta(seconds=60)
ALLOWED_LATENESS = timedelta(minutes=60)
def at(hms):
return datetime.strptime(hms, "%H:%M:%S")
def window_start(event_time):
minute = event_time.minute - event_time.minute % 5
return event_time.replace(minute=minute, second=0)
def run(events):
totals, fires, output, late = {}, {}, {}, []
watermark = datetime.min
for event_id, event_time, amount in events:
start = window_start(event_time)
if watermark >= start + WINDOW + ALLOWED_LATENESS:
late.append(event_id)
continue
totals[start] = totals.get(start, 0) + amount
if start in fires:
fires[start] += 1
output[start] = totals[start]
watermark = max(watermark, event_time - WATERMARK_LAG)
for open_start in list(totals):
end = open_start + WINDOW
if watermark >= end and open_start not in fires:
fires[open_start] = 1
output[open_start] = totals[open_start]
if watermark >= end + ALLOWED_LATENESS:
del totals[open_start]
return output, fires, late
events = [
("E1", at("09:02:14"), 100),
("E2", at("09:03:55"), 200),
("E5", at("09:06:01"), 40),
("E3", at("09:02:14"), 50),
("E6", at("10:06:30"), 60),
("E4", at("09:02:14"), 75),
]
output, fires, late = run(events)
windowtotalfires
[09:00, 09:05)3502
[09:05, 09:10)401

The function returns late as ['E4']. Window [10:05, 10:10) holds E6's 60 and has not fired, because the watermark stands at 10:05:30.

Every re-fire replaces the last one

Each firing emits the window's full current total, and a later firing supersedes an earlier one. A sink that appends keeps both rows for 09:00, and a dashboard that sums the table reports 650. The sink has to upsert on the window key: the replace-not-append rule from idempotent reruns, applied to streaming output.
Append sink
windowtotal
09:00-09:05300
09:00-09:05350
Upsert on window
windowtotalfires
09:00-09:053502
Multiple Choice

A team's sink appended every firing to revenue_firings, one row per firing with a fired_at timestamp. Late refunds can lower a total on re-fire. Which aggregation returns the current total per window?

What an hour of lateness costs

Every window's state now outlives its firing by an hour of watermark progress, so each key carries the open window plus every window that fired in the previous hour. A revenue job keyed by city across 600 cities multiplies that by 600. Readers also see a number that may still change: the 09:00 total is final only once the watermark moves an hour past 09:05.
12
fired windows held per key
7,200
fired windows held across 600 cities
Where very late events go
  • ▸Route them to late_events; a silent drop hides how often lateness runs out.
  • ▸Chart the daily count of late_events next to the window totals.
  • ▸Let the batch rerun of the last 7 days fold them into the curated trips table.
  • ▸If late_events keeps growing, measure skew again before raising allowed lateness.
Multiple Choice

A job uses 5-minute windows, a 60-second lag and 60 minutes of allowed lateness. The watermark reads 14:40:00. Three events then arrive, stamped 14:03:00, then 15:06:00, then 14:04:00. What happens to window [14:00, 14:05)?

PUTTING IT ALL TOGETHER

> Picture Uber's earnings pipeline during the fare rename. Version 3 of trip_completed passes the registry's FULL check because every change carries a default. While the 3 consumers switch over the following month, the streaming earnings job keeps summing 5-minute windows by ended_at, with a 60-second watermark lag and 60 minutes of allowed lateness, and upserts each firing into the earnings table. Trips uploaded after their window's lateness runs out land in late_events, and the daily rerun of the last 7 days folds them into trips.

FULL compatibility is what lets the 4 expand-contract phases ship on separate team schedules.
The earnings job buckets by ended_at, because driver pay follows when a trip happened.
The watermark lag comes from measured skew between processed_at and ended_at.
Allowed lateness re-fires windows, so the sink upserts on window key: the replace-not-append rule from idempotent reruns.
Very late trips reach late_events like a dead letter queue, and the 7-day batch rerun repairs the curated table.
KEY TAKEAWAYS
A schema registry stores versions and schema IDs per subject, and its compatibility mode decides which new versions it accepts.
BACKWARD means consumers deploy first, FORWARD means the producer deploys first, and FULL lets either side go first.
Expand-contract turns a breaking rename into 4 compatible deploys: expand, migrate, switch, contract.
Event time answers questions about the world; processing time answers questions about the pipeline.
A watermark is the engine's assertion that no earlier events remain, and a window fires when the watermark reaches its end.
Allowed lateness keeps a fired window's state for a stretch of watermark progress, so late events re-fire it and the sink must upsert.

Registries, expand-contract and watermarks: how pipelines survive renamed fields and fares that arrive an hour late.

Category
Pipeline Architecture
Difficulty
intermediate
Duration
32 minutes
Challenges
12 hands-on challenges

Topics covered: Schema Registries: Where They Live, The Expand-Contract Pattern, Event Time Versus Processing Time, Watermarks: The Engine's Promise, 1-Hour Allowed Lateness

Lesson Sections

  1. Schema Registries: Where They Live (concepts: Schema Evolution)

    A schema registry is a small service that stores every version of every event schema and checks each proposed version against the versions before it. In the beginner lesson, a producer changed its event shape and a consumer broke at runtime. A registry moves that failure to deploy time, where it costs a red build instead of a broken dashboard. What the registry stores The registry also changes what travels on the topic. Each message carries a short schema ID ahead of its payload, in place of the

  2. The Expand-Contract Pattern (concepts: Schema Evolution)

    The 4 phases Expand: add beside the old field Migrate: give history the new fields Switch: consumers move one by one Contract: remove what nobody reads

  3. Event Time Versus Processing Time (concepts: Stream Processing)

    Count those 6 trips per hour and the 2 clocks disagree about every hour. T3 and T6 both ended in the 08:00 hour, yet T3 reached the pipeline at 09:12 and T6 at 10:02. Which clock answers which question

  4. Watermarks: The Engine's Promise (concepts: Late-Arriving Data)

    A daily batch job knows yesterday is finished when the last file lands. A streaming job that sums revenue in 5-minute event-time windows gets no such signal, and T6 showed that a trip can surface 67 minutes after it ended. Before the job publishes a total for 09:00 to 09:05, it needs a rule for deciding it has waited long enough. A watermark is that rule. Max event time seen, minus a lag Most engines compute a heuristic watermark from the data itself: the largest event time seen so far, minus a

  5. 1-Hour Allowed Lateness (concepts: Late-Arriving Data)

    Lateness runs on the watermark The window logic in plain Python The function below holds the whole policy: a 5-minute window, a 60-second lag and 60 minutes of allowed lateness. For each event it checks whether the window's lateness has run out, updates the total, re-fires a window that already fired, then advances the watermark and fires or discards windows. Every re-fire replaces the last one Each firing emits the window's full current total, and a later firing supersedes an earlier one. A sin