Contracts on a Legacy Pipeline

Concepts covered: Schema Evolution

Greenfield contracts are easy. Contracts on a legacy pipeline that has run for four years and has dozens of unknown consumers are hard. The mistake most teams make is treating the rollout as a one-shot migration: write the contract, declare the producer compliant, declare consumers responsible for catching up. The mistake produces breakage and erodes the credibility of the contract program. The disciplined rollout treats existing consumer behavior as the starting contract, evolves toward the desired contract over time, and uses the same shape that semver software libraries have used for decades. The exercise below walks through the rollout for a customer events stream that is the lifeblood of three downstream consumers and an unknown number of analyst queries. The legacy character of the pipeline is what makes the rollout interesting. A greenfield rollout has the luxury of starting from a clean schema and an enumerable consumer set. A legacy rollout has neither. The discipline below is what compensates for the missing luxury, and the principles transfer to any production system that has been running long enough that the original authors have moved on and the original assumptions have drifted.

The Starting State

Step 1 is to document the current contract. The first contract is not the contract the team wants; it is the contract the producer is currently delivering, including the drift. The team writes it by inspecting recent production data: every field that has appeared in the last 30 days, every type variation seen, every value range observed. The result is messy and accurate. The contract version is 1.0.0, and it captures reality, not aspiration.

# Contract version 1.0.0: documents what is, not what should be
contract:
  name: customer_events
  version: 1.0.0
  status: documented_legacy
  schema:
    primary_key: event_id
    fields:
      - { name: event_id,        type: string,    nullable: false }
      - { name: customer_id,     type: string,    nullable: true,
          observed_null_rate: 0.008 }
      - { name: event_type,      type: string,    nullable: true,
          observed_values: [signup, login, purchase, churn,
                            page_view, click, error] }
      - { name: ts,              type: union[timestamp, string],
          nullable: false,
          notes: 'producers send both ISO-8601 strings and epoch ms' }
      - { name: amount,          type: union[decimal, string],
          nullable: true,
          notes: 'string for older clients; decimal for newer' }
  guarantees:
    freshness: 'best effort, typically <= 30 minutes'
    volume: 'no enforced bound; observed 1.5 to 4M per day'
  consumers_known:
    - leadership_dashboard
    - churn_model
    - billing_aggregator

Step 2 is to inventory every consumer. The three named consumers are easy. The unknown analyst queries are not. Lineage tooling helps: parsing recent query logs against the table catches most analyst queries; manual outreach catches the rest. The output is a list of every reader, ranked by frequency, with an owner per consumer. Step 3 adds quality gates in observation-only mode. The gates run, log results, and do not block. After two to four weeks of observation, the team has data on which gates would have fired and tunes against it. Once tuned, gates switch from observation to enforcing. Skipping the observation phase is the most common rollout mistake.

# Gates start in observation mode
quality_gates:
  - name: row_count_within_baseline
    layer: raw
    mode: observe   # later: 'block'
    threshold: 0.50_to_2.00_of_dow_baseline
  - name: customer_id_null_rate
    layer: raw
    mode: observe
    threshold: 0.01
  - name: amount_distribution_z_shift
    layer: curated
    mode: observe
    block_threshold: 5.0
    warn_threshold: 3.0
  - name: event_type_accepted_values
    layer: raw
    mode: observe
    threshold: enum_present_in_recent_history

Step 4 plans the cleanups as versioned changes. Legacy problems (union types on ts and amount, unwhitelisted event_types, ambiguous nulls) are each a versioned change. Renaming ts to event_timestamp with strict ISO-8601 is a breaking change; it ships in version 2.0.0 with a 90-day notice. Whitelisting accepted_values for event_type is backward-compatible if the producer commits to not emitting new values without a contract update; it ships as minor 1.x. Each change has a migration plan naming affected consumers and their actions. Cleanups happen one at a time, not as a single rewrite.

Step 5 migrates consumers behind the versions. Each consumer migrates to the next major version on its own schedule within the announced notice period. The producer maintains both old and new versions during the transition window; dual-publishing is more expensive than single-publishing, and that maintenance burden falls on the producer. The benefit is that no consumer breaks in production; each migration is a deliberate code change with tests, not a surprise in an incident channel. Some consumers migrate in a week; others take the full notice period. Both are acceptable; a unilateral cutover is not. The producer tracks migration progress with a simple consumer-version query against the contract registry.

SELECT contract_version,
       COUNT(DISTINCT consumer_team) AS pinned_consumers,
       MIN(last_seen) AS earliest_pin,
       MAX(last_seen) AS latest_pin
FROM contract_registry.pins
WHERE contract_name = 'customer_events'
  AND last_seen >= CURRENT_DATE - 7
GROUP BY contract_version
ORDER BY contract_version;

Producer publishes:                                                                  
                                                                                     
  customer_events 1.x  -+----------+----------+----------+----------+ deprecation    
  customer_events 2.x  -+----------+----------+----------+----------+ live           
  customer_events 3.x  -+ pre-announce ----+----------+ live --      ~                
                                                                                     
  Day 0   Day 30   Day 60   Day 90   Day 180   Day 270                               
                                                                                     
  Consumer A pins 1.0 ---> migrates to 2.0 around Day 60 ---> migrates to 3.0 by 270 
  Consumer B pins 1.0 ---> migrates to 2.0 by Day 90 ---> stays on 2.x               
  Consumer C pins 2.0 ---> migrates to 3.0 by Day 270                                

Step 6 locks in the new state. Once consumers are migrated and the legacy version is deprecated, the producer flips the gates from observation-only to enforcing on the new contract. CI now rejects producer changes that violate the contract. Schema drift becomes impossible; incidents that used to come every six to eight weeks become rare. The rollout is complete when the producer can commit to the contract as a CI-enforced guarantee, not documentation alone.

Before The Rollout

• Wrong-number incident every six to eight weeks
• Three named consumers; unknown analyst surface
• Schema drifts continuously; producers add fields without notice
• Quality is best-effort; no SLA possible
• Rolling back a producer change requires consumer-by-consumer triage

After The Rollout

• Wrong-number incidents are rare; most are caught in CI
• Every consumer is registered; lineage covers analyst queries
• Schema evolves through versioned changes with notice
• Operational and quality SLAs are stated and tracked
• Producer can roll back any version because all consumers are pinned

Do

• Treat the first contract as a documentation of reality, not aspiration
• Use semver explicitly; minor for additive, major for breaking, with notice
• Maintain old and new versions in parallel during the migration window

Avoid

• Cut over consumers unilaterally; the rollout is a negotiation, not a mandate
• Skip observation mode; thresholds need real data before they can be enforced
• Treat the rollout as one project; treat each version bump as its own change

Contracts and quality become a system only when the producer can roll back any version because every consumer is pinned. The concise statement of the rollout principle: a legacy rollout earns its trust by treating consumers as parties to a negotiation, not as inventory to be migrated, structured by versions and notice periods. The trust accumulates over the months in which no consumer is broken by surprise.

About this lesson

This section is part of the Data Quality: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.