Alerting That Stays Useful

A monitor decides pass or fail. An alert decides who hears about a failure, and how fast. Keeping the 2 decisions apart lets every monitor record its result while only a few results interrupt a person.

3 severities

SeverityWhat happensUse it whenIn orders_daily
PageRings the on-call engineer at any hour until acknowledgedA consumer will be harmed soon and only a person can stop itFreshness fails at 06:00
TicketOpens a work item handled in working hoursA person must act, but a day's wait harms no one3 slow nights in a row
RecordWrites to a dashboard or channel, interrupting nobodyContext for later, or a problem already handledA retry that succeeded

Choosing between a page and a ticket comes down to 2 durations: how long until a consumer is harmed, and how long a fix takes. At 06:00 the dashboard is 2 hours away and a rerun takes about 40 minutes, so a 06:00 page leaves room to finish the repair.

Page on what the consumer feels

Pages work best on symptoms, the effects a consumer would notice: data that is late or missing, or numbers that are wrong. Causes such as a refused connection or a slow task fit tickets and records better, since retries absorb many of them. A symptom page also stands in for the several cause alerts a single incident sets off.

MonitorFiresGoes to
Run failureWhen a task runs out of retriesTeam channel and a ticket
Volume gateBefore publishTeam channel and a ticket; the load is held
Freshness06:00 UTCA page to the on-call engineer
Duration over 80 minutesAfter the third slow night in a rowA ticket
Retry that succeededAny timeA record only

With this routing, the extract outage in the run for March 15 produces a single page. The run-failure post from 02:15 is already in the channel when the freshness page arrives at 06:00, so the engineer starts with the cause named.

A different pipeline feeds a report that opens at 08:00. A rerun takes 3 hours. Its only page is a freshness check at 06:00. What should change?

Noise wears out the on-call

Each page that turns out to need no action teaches the on-call engineer that pages can wait. After enough of them, a real page waits too. Picture a team whose alert history for a week looked like this:

Most of the rest were retries that later succeeded and volume ranges drawn too tight. Demoting the first to records and widening the second removes the noise and hides no real incident.

What a page should say

Hard to act on
  • Task failed: extract_orders
  • No date, so the reader hunts for the run
  • No impact, so the reader cannot judge urgency
  • No link, so the reader starts from a blank screen
Ready to act on
  • orders_daily: no rows in daily_revenue for 2026-03-15 at 06:00 UTC
  • Impact: finance dashboard shows stale revenue from 08:00 UTC
  • Cause so far: extract_orders failed at 02:14 after 3 retries
  • Runbook: runbooks/orders_daily, freshness section
Do
  • Page only when a consumer will be harmed and a person must act now
  • Give every page the date and impact, plus a link to the runbook
  • Demote any alert that fired twice with no action taken
Avoid
  • Page on a retry that went on to succeed
  • Page more than once for a single incident
  • Route alerts to a shared inbox nobody owns

For the third night in a row, orders_daily ran over 80 minutes. Each run still finished before 03:40. How should this alert be routed?

About this lesson

This section is part of the Pipeline Operations: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.