Day-One Monitoring

Signals sit unread until something checks them. A monitor is a scheduled check that reads a signal and returns pass or fail. A new pipeline needs only the handful of monitors that between them catch its most common breakages, and each one should catch something the others cannot.

3 monitors to ship with

orders_daily with its day-one monitors. Orders move left to right; check_volume must pass before load_revenue publishes the day, and the dashboard reads daily_revenue at 08:00 UTC.
  • orders replica (source, operational database)
  • extract_orders (transform, orchestrator); retries: 3; alerting: run failure
  • transform_orders (transform, orchestrator); retries: 3; alerting: run failure
  • stg_daily_revenue (storage, staging table)
  • check_volume (quality gate, orchestrator); on error: hold the load outside 1.0M to 1.4M
  • load_revenue (transform, orchestrator); retries: 3; idempotency: replace the day's rows; alerting: run failure
  • daily_revenue (storage, warehouse table); freshness: yesterday by 06:00 UTC
  • finance dashboard (consumer, dashboard, 08:00 UTC)
  • orders replica -> extract_orders
  • extract_orders -> transform_orders
  • transform_orders -> stg_daily_revenue
  • stg_daily_revenue -> check_volume
  • check_volume -> load_revenue
  • load_revenue -> daily_revenue
  • daily_revenue -> finance dashboard

For the volume check to act as a gate, transform_orders now writes to a staging table, stg_daily_revenue, with the same columns as daily_revenue. A fourth task, check_volume, counts the orders in that table. load_revenue copies the day across only when the count passes.

What each monitor catches alone

Each column below imagines a single monitor as the only one installed. Each row is a real way orders_daily breaks.

What brokeRun failure aloneFreshness aloneVolume alone
`extract_orders` fails after 3 retriesFires at 02:15Fires at 06:00Silent: check never runs
A deploy leaves the schedule pausedSilent: no run, no failureFires at 06:00Silent: check never runs
`transform_orders` hangs for 6 hoursSilent: still runningFires at 06:00Silent: check never runs
Replica lag: 312,880 orders, run succeedsSilent: run succeededSilent: the date is presentFires before publish

Freshness fires in 3 of the 4 rows, which makes it the monitor to keep if only a single one were allowed: it watches what the consumer will see, so it notices failures no task reported. Run failure fires 3 hours and 45 minutes sooner in the first row and names the broken task. Volume is the only check that judges the data; without it, the quarter-sized March 10 load reaches the dashboard.

A Friday deploy leaves the orders_daily schedule paused. The team installed only the run-failure and volume monitors. When does anyone find out?

Writing the data checks

Schema
  • daily_revenue: revenue_date DATE PK, region VARCHAR PK, revenue DECIMAL(18,2), order_count BIGINT, loaded_at TIMESTAMP

The freshness query runs at 06:00 UTC against the table above and fails when days_behind is greater than 1.

SELECT
  max(revenue_date) AS latest_date,
  date_diff('day', max(revenue_date), current_date) AS days_behind
FROM daily_revenue

The volume query runs against the staging table after transform_orders. The orchestrator writes the run's date into it, so a rerun for March 10 counts March 10.

SELECT
  sum(order_count) AS orders
FROM stg_daily_revenue
WHERE revenue_date = DATE '2026-03-10'

Set the range wide, from a few weeks of history; 1.0 to 1.4 million covers every normal March night. A 1.9-million-order sale day will trip it on good data, and the owner raises the ceiling for that date.

A volume check is written as below. On March 12, the owner reruns the pipeline for March 10. Which day's orders does this check count?

SELECT
  sum(order_count) AS orders
FROM stg_daily_revenue
WHERE revenue_date = current_date - INTERVAL '1' DAY

About this lesson

This section is part of the Pipeline Operations: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.