Reading a Real Pipeline Diagram
Concepts covered: Pipeline Monitoring
The diagram below is the retailer's order pipeline as it runs today, with the operating facts a production diagram carries and a sketch leaves off. Reading it is a procedure, and applied to a 40-node diagram at a new job it tells you in 20 minutes what a month of incidents would otherwise teach.
- kafka_orders (source, Kafka); freshness: events visible within 60 s; alerting: consumer lag > 5 min
- pg_catalog (source, Postgres + Debezium); alerting: connector state != RUNNING
- s3_raw (bronze, S3 Parquet); retries: 3; backoff: 5 min; idempotency: overwrite hourly prefix
- curated_orders (silver, Snowflake + dbt); retries: 2; backoff: 10 min; backfill: rerun interval, trailing 3 h; idempotency: delete and insert by order_hour
- dq_gate (quality gate, dbt tests); on error: block downstream; alerting: inputs fresh within 60 min; category null rate < 0.1%; row count = curated
- finance_mart (gold, Snowflake + Looker); freshness: yesterday complete by 08:00
- feast_features (consumer, Feast); freshness: 15 min
- kafka_orders -> s3_raw
- pg_catalog -> s3_raw
- s3_raw -> curated_orders
- curated_orders -> dq_gate
- dq_gate -> finance_mart
- dq_gate -> feast_features
- Sources: kafka_orders, pg_catalog
- Raw: s3_raw
- Curated: curated_orders, dq_gate
- Serving: finance_mart, feast_features
Ends first, then the fan-in
Start at the 2 ends, because they set the requirements. Sources decide arrival behaviour: Kafka delivers within seconds and replays from any offset inside its 7 days of retention, while the Debezium connector follows the Postgres write-ahead log and fails its task when it loses its replication slot. Consumers decide freshness: the finance mart promises yesterday complete by 08:00, the feature store promises 15-minute freshness. Then find where edges converge. Here orders meet the catalog at curated_orders, the role orders_enriched played in section 2: its cadence is bounded by its slowest input, and every consumer inherits that bound.
Cadence per edge
An edge carries no words, so each one gets 2 questions: what moves along it and how often. Curated to the mart is hourly aggregation; curated to the feature store is a 15-minute refresh reading an hourly table, so 3 of every 4 refreshes return nothing new. A cadence mismatch across an edge is a finding, and the next section resolves this one.
Failure behaviour per node
For each node, 3 questions. Does it retry, and is its write safe to retry? The raw landing overwrites an hourly S3 prefix, so 3 retries leave 1 copy. The curated load deletes and inserts by order_hour in 1 transaction, so a retry after a timed-out commit removes its own earlier rows before loading. A load that appended instead would commit 40,000 rows, lose the acknowledgement, and commit 40,000 more on attempt 2, with every task green and finance seeing double revenue at 08:00. Does a failure block downstream or let stale data through? The gate blocks, so a bad hour never reaches the mart, at the price of a frozen dashboard. Who is paged? The monitorAlert on each node names the condition, and the owner column names the person.
| Node | Cadence | On failure | Owner |
|---|---|---|---|
| kafka_orders | continuous | lag alert at 5 min, replay within 7 days | Checkout team |
| pg_catalog | change stream, flushed every 15 min | connector-state alert, no automatic recovery after slot loss | Catalog team |
| s3_raw | hourly | 3 retries 5 min apart, overwrite prefix | Data platform |
| curated_orders | hourly | 2 retries, delete and insert, pages platform on-call | Data platform |
| dq_gate | hourly | blocks both consumers, pages platform on-call | Data platform |
| finance_mart | hourly | 2 retries, dashboard shows stale hours | Finance analytics |
| feast_features | 15 min | 1 retry, model serves last good values | Recommendations |
Ownership at the layer boundary
Ownership changes hands where the layer changes. The checkout team owns the topic and its schema, the catalog team owns the Postgres tables, the data platform owns raw through the gate, and each consuming team owns its serving table. The boundary between layers is where the contract lives: the checkout team may add fields to orders.v1 but may not rename order_id, and the platform may change how curated_orders is built but not what a row means. A diagram without owners turns every cross-boundary incident into 20 minutes of finding out who to call.
At 02:00 the Debezium connector for the catalog loses its replication slot and its task fails. Reading the diagram above, what does the finance mart show at 09:00, and who was paged?
- Hours 02 to 08 present with category null for new products; nobody paged, because every task succeeded
- The mart frozen at hour 01; the catalog team paged by the connector alert at 02:00, and the platform on-call paged when the gate blocked hour 02
- The mart up to date, because the join reads the SCD2 catalog and existing products still resolve
- Hours 02 to 08 missing after Airflow retried curated_orders twice and gave up, paging only platform
About this lesson
This section is part of the Pipeline Anatomy: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.