Pipelines as Products

Concepts covered: Data Quality & Validation

The intermediate lesson drew pipelines as boxes and arrows: sources on the left, a curated layer in the middle, consumers on the right. Those diagrams answer where data moves. A product framing asks a second question of every box on the right edge: who reads this, and what did we promise them? A pipeline is a product when it has a named consumer and a written contract, and when someone has planned how it will be retired. A pipeline with none of those is a script that happens to run on a schedule, and the first person to notice it broke is a stranger on another team.

Name the consumer

Take a curated table of music listening sessions, one row per user per session, read by 3 teams. Each reads it for a different reason, so each breaks in a different way.

Without this list, every design decision is a guess. Should the pipeline ship a partial day at 06:00 or a complete day at 09:00? Recommendations wants partial and royalties wants complete. The answer is 2 outputs: a listening_sessions_early snapshot at 06:00 with recommendations as its only consumer, and the final table at 09:00 for everyone else.

Write the contract

A contract is the promise in writing: what a row means, which columns exist with which types, when the table is ready, how complete it is, how it may change, and who gets paged when any of that fails. It fits on one screen, and each consumer signs off on it before the first row lands. Here is the contract for the final table.

product: curated.listening_sessions
owner: listening-data-team
grain: one row per user per listening session
consumers:
  - royalty-reporting
  - yearly-recap
ready_by: "09:00 UTC"
completeness: at least 99.5 percent of the sessions landed in raw for that day
columns:
  user_id: bigint, never null
  session_start: timestamp, never null
  ms_played: bigint
  event_date: date, partition key
change_policy: additive only, 30 days notice before any removal or rename
paged: listening-data-oncall

Each line of the contract maps to a check the pipeline runs on itself. The ready_by and completeness lines become the query below, run at 09:00. It fails if the curated table holds under 99.5% of the sessions that landed in raw.listening_sessions for yesterday, or if yesterday's latest session started before 23:00, a sign the day was cut off. The columns line becomes a check on names and types.

SELECT
  count(*) >= 0.995 * (
    SELECT
      count(*)
    FROM raw.listening_sessions
    WHERE event_date = current_date - INTERVAL '1' DAY
  ) AS is_complete,
  max(session_start) >= CAST(current_date - INTERVAL '1' DAY AS timestamp) + INTERVAL '23' HOUR AS reaches_day_end
FROM curated.listening_sessions
WHERE event_date = current_date - INTERVAL '1' DAY

Define the deprecation path

Every table is eventually replaced. The product framing plans that replacement on day 1, because the consumer list is the only way to know who the replacement will hurt. The path has 4 steps, and the contract's consumers field drives the 3rd.

• Ship the replacement under a new name, listening_sessions_v2, with its own contract
• Run both for a fixed window, 30 days, and publish the differences between them
• Contact each listed consumer and confirm they have moved; the list is the migration checklist
• Announce a final date for the old table, then drop it on that date, not 'soon'

A team runs a nightly job that writes orders_daily. It has run for 3 years, 6 dashboards query it, and nobody on the team can say which 6. Which single change moves it furthest toward being a product?

• Add a _v2 suffix to the table name so a future replacement has somewhere to go
• Query the warehouse's access logs, list the 6 dashboards and their owners, and record them with the table
• Add a row-count check so the team is paged when the nightly write produces 0 rows
• Move the job to a 09:00 schedule so it has time to finish before the dashboards refresh

About this lesson

This section is part of the Pipeline Anatomy: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.