A First Runbook
A runbook is the written procedure for a single alert. Its reader is the engineer the alert has reached, who may never have opened the pipeline's code. Its 5 sections run in the order that reader needs them: symptom and impact first, then diagnosis and response, with escalation last. Here is the runbook for the orders_daily freshness page.
Runbook: orders_daily freshness page
Symptom
The freshness monitor found no rows in daily_revenue for yesterday at 06:00 UTC. A normal run finishes by roughly 02:40.
Impact
Finance opens the revenue dashboard at 08:00 UTC. Until the load lands, the newest day it shows is the day before yesterday. No other team reads this table.
Diagnosis
- Step 1. Open the orders_daily run for yesterday's date in the orchestrator. If no run exists, the schedule is paused or did not trigger: go to Response A.
- Step 2. If the run is still going, open its trace and compare each task with its usual minutes: extract 24, transform 10, load 6. Go to Response B.
- Step 3. If extract_orders, transform_orders or load_revenue failed, read the ERROR line from its last attempt. connection refused or timeout: the replica was unreachable, go to Response C. column not found or a type error: the source schema changed, go to Escalation and do not rerun.
- Step 4. If check_volume failed, run the volume query for yesterday's date and note the count. Go to Response D.
Response
- A. Unpause the schedule, then trigger a run with --date set to yesterday.
- B. If a task has run past 3 times its usual minutes, stop the run and start it again with --date. Otherwise check again at 06:30.
- C. Run SELECT 1 against orders-replica port 5432. If it answers, rerun with --date; load_revenue replaces the day's rows, so the rerun is safe. If refused, go to Escalation.
- D. Under 1.0 million usually means replica lag. Measure it on the replica with SELECT date_diff('minute', max(created_at), current_timestamp) FROM orders and rerun once it is under 10 minutes. Over 1.4 million on a known sale day: raise that date's ceiling, then rerun. Over 1.4 million otherwise: likely duplicated source rows, so page the orders database on-call and do not rerun.
- Finish every response by running the freshness query. days_behind must be 1.
Escalation
- No acknowledgement within 15 minutes: the secondary on-call is paged automatically.
- Not fixed by 07:30 UTC: tell the finance channel that yesterday's revenue is late, and when the next update will come.
- Replica down for over 30 minutes, or a schema change: page the orders database on-call.
- Owner: revenue data team. Last followed end to end on 2026-03-01.
At 06:00 on March 21 the freshness page fires. In the run for March 20, extract_orders and transform_orders succeeded. Then check_volume failed at 02:12, so load_revenue never started. The volume query returns 402,115. What does the runbook have you do?
- Step 3 applies because a task failed, so confirm the replica accepts connections and rerun at once.
- Step 4 applies, and Response D says to check replica lag, wait until it is under 10 minutes, then rerun.
- Step 4 applies, and the low count is real, so lower the floor to 400,000 and rerun the load.
- Step 1 applies because no data was published, so unpause the schedule and trigger a new run.
Writing for a stranger at 3 a.m.
- Name the exact query, command or screen; 'check the logs' names none of them
- Give each check its possible results and the next step for every result
- Say when a rerun would make things worse, as well as when it helps
- Have an engineer new to the pipeline follow it in working hours, and record the date
- Update it after each incident, while the missing steps are fresh
A draft runbook's diagnosis section reads: 'Check the logs and fix the issue.' Which rewrite helps an engineer who has never seen the pipeline?
- Add a paragraph on how the pipeline was designed, so the reader understands the code before acting.
- Rerun the pipeline first, and read the logs only if the rerun fails as well.
- Contact the pipeline's author, who knows the code best and can find the problem fastest.
- Name the log line to read, list the errors it may show, and give the next step for each.
About this lesson
This section is part of the Pipeline Operations: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.