Pipeline Operations: Intermediate

Five pillars of observability, lineage, cost attribution, and CI before production

Five pillars of observability, lineage, cost attribution, and CI before production

Category

Pipeline Architecture

Difficulty

intermediate

Duration

32 minutes

Challenges

0 hands-on challenges

Topics covered: The Five Pillars of Observability, Lineage and Blast Radius, Cost Attribution, CI/CD for Pipelines, Instrumenting a Pipeline E2E

Lesson Sections

1. The Five Pillars of Observability (concepts: paFivePillars, paFreshness, paSchemaMonitor, paDistributionCheck)

   The five pillars framework, popularized by Monte Carlo and Barr Moses, names the kinds of signal a mature data observability practice tracks. The pillars are freshness, volume, schema, distribution, and lineage. They are not a checklist of monitors. They are a vocabulary for naming where the eyes and the gaps are. A pipeline well covered on freshness and volume but blind on distribution will fail in a particular family of ways; a pipeline blind on lineage will fail differently. The framework let

2. Lineage and Blast Radius (concepts: paLineage, paBlastRadius)

   Lineage is the graph of which datasets depend on which others. Read it forward and it answers 'who consumes this table.' Read it backward and it answers 'what produced this column.' Both directions matter operationally. Without lineage, an engineer changing a column has no way to know who breaks; without lineage, an engineer debugging a wrong number has no way to know which pipeline to inspect first. Lineage is the difference between a five-minute fix and a five-hour fix. Forward Lineage and Bla

3. Cost Attribution (concepts: paCostAttribution, paQueryTags)

   Most data teams do not know what their pipelines cost until somebody asks. The bill arrives as a single number from Snowflake or BigQuery or Databricks; it does not break down by pipeline. Without attribution, the cost conversation is impossible: nobody can say which pipelines should be optimized, which ones can be retired, or which ones are growing fastest. The fix is query tagging, which threads a pipeline identifier through every query the warehouse runs. The pattern is universal across cloud

4. CI/CD for Pipelines (concepts: paCiCd, paSlimCi)

   Application engineers ship changes through CI/CD: a pull request runs unit tests, integration tests, and a deploy step. Pipeline changes are different in two ways. First, the data is part of the test surface; a transform is correct only if it produces the right output on real-shaped input. Second, the pipeline operates on data the test environment may not have. Both differences shape what CI for pipelines looks like, and both are misunderstood by teams that try to import application CI patterns

5. Instrumenting a Pipeline E2E (concepts: paInstrumentation, paOperationalSequencing)

   Take a real pipeline and walk it through the operational instrumentation. The pipeline is fct_orders, a daily aggregation that pulls from a Postgres replica, transforms in Snowflake via dbt, and feeds a Looker dashboard, a daily revenue email, and an ML feature store. The pipeline runs cleanly today. It has no observability beyond the orchestrator's run status and no cost visibility. The exercise is to bring it to a level-three operational state in a sequence that delivers value at each step. St