Alerting That Stays Useful
A monitor decides pass or fail. An alert decides who hears about a failure, and how fast. Keeping the 2 decisions apart lets every monitor record its result while only a few results interrupt a person.
3 severities
| Severity | What happens | Use it when | In orders_daily |
|---|---|---|---|
| Page | Rings the on-call engineer at any hour until acknowledged | A consumer will be harmed soon and only a person can stop it | Freshness fails at 06:00 |
| Ticket | Opens a work item handled in working hours | A person must act, but a day's wait harms no one | 3 slow nights in a row |
| Record | Writes to a dashboard or channel, interrupting nobody | Context for later, or a problem already handled | A retry that succeeded |
Choosing between a page and a ticket comes down to 2 durations: how long until a consumer is harmed, and how long a fix takes. At 06:00 the dashboard is 2 hours away and a rerun takes about 40 minutes, so a 06:00 page leaves room to finish the repair.
Page on what the consumer feels
Pages work best on symptoms, the effects a consumer would notice: data that is late or missing, or numbers that are wrong. Causes such as a refused connection or a slow task fit tickets and records better, since retries absorb many of them. A symptom page also stands in for the several cause alerts a single incident sets off.
| Monitor | Fires | Goes to |
|---|---|---|
| Run failure | When a task runs out of retries | Team channel and a ticket |
| Volume gate | Before publish | Team channel and a ticket; the load is held |
| Freshness | 06:00 UTC | A page to the on-call engineer |
| Duration over 80 minutes | After the third slow night in a row | A ticket |
| Retry that succeeded | Any time | A record only |
With this routing, the extract outage in the run for March 15 produces a single page. The run-failure post from 02:15 is already in the channel when the freshness page arrives at 06:00, so the engineer starts with the cause named.
A different pipeline feeds a report that opens at 08:00. A rerun takes 3 hours. Its only page is a freshness check at 06:00. What should change?
- Page when a task fails, because a 06:00 page leaves 2 hours for a fix that needs 3.
- Keep the 06:00 page and add a ticket for each failure, so causes are reviewed the next day.
- Move the freshness check to 07:30, so it fires closer to the moment the report opens.
- Turn the freshness page into a ticket, since paging at 06:00 wakes people for nothing.
Noise wears out the on-call
Each page that turns out to need no action teaches the on-call engineer that pages can wait. After enough of them, a real page waits too. Picture a team whose alert history for a week looked like this:
- 41
- 3
Most of the rest were retries that later succeeded and volume ranges drawn too tight. Demoting the first to records and widening the second removes the noise and hides no real incident.
What a page should say
- Hard to act on
- Task failed: extract_orders
- No date, so the reader hunts for the run
- No impact, so the reader cannot judge urgency
- No link, so the reader starts from a blank screen
- Ready to act on
- orders_daily: no rows in daily_revenue for 2026-03-15 at 06:00 UTC
- Impact: finance dashboard shows stale revenue from 08:00 UTC
- Cause so far: extract_orders failed at 02:14 after 3 retries
- Runbook: runbooks/orders_daily, freshness section
- Do
- Page only when a consumer will be harmed and a person must act now
- Give every page the date and impact, plus a link to the runbook
- Demote any alert that fired twice with no action taken
- Avoid
- Page on a retry that went on to succeed
- Page more than once for a single incident
- Route alerts to a shared inbox nobody owns
For the third night in a row, orders_daily ran over 80 minutes. Each run still finished before 03:40. How should this alert be routed?
- Page the on-call engineer, because 3 slow nights in a row put the 06:00 promise at risk.
- Record it on the dashboard only, because the data arrived on time and needs no person.
- Open a ticket for the owning team, because a person should look but a day's wait harms no one.
- Page the source database team, because slow runs usually start in the system being read.
About this lesson
This section is part of the Pipeline Operations: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.