Pipeline Operations: Beginner
- print("done") writes to a terminal nobody watches. Seen from anywhere else, a failed night and a good night look the same.
- date.today() ties the processed day to the clock, so once Monday has passed there is no way to ask for Monday again.
- warehouse.insert appends. Two runs for the same day write that day's totals twice, the duplicate-row bug from Idempotent Reruns.
- Nothing records the row count or the run time, so nobody can say whether a night's 312,880 orders is normal or a warning.
orders.py fails at 02:00 on Tuesday. The Wednesday 02:00 run succeeds. At 09:00 on Wednesday an analyst spots a gap and runs the script once more by hand. What does daily_revenue now hold for Monday and Tuesday?
Operable means a stranger can run it
- Started by cron on a single server
- Processes whatever day the clock says
- Appends on every run
- Reports success to a terminal
- Breakage found by the people reading dashboards
- Scheduled and retried by an orchestrator
- Takes the date to process as an input
- Replaces the day's rows, so reruns are safe
- Sends its logs and metrics somewhere searchable, along with the status of each run
- Monitors watch the output and alerts reach an owner, with a runbook to follow
The rest of this lesson works on orders_daily, the same job rebuilt as a DAG under an orchestrator. Its facts stay fixed in every section:
| Property | Value |
|---|---|
| Pipeline | `orders_daily` |
| Schedule | Daily at 02:00 UTC, processing the previous day |
| Tasks | `extract_orders`, `transform_orders`, `load_revenue` |
| Source | Orders database, read through a replica |
| Output | `daily_revenue`, a row per region per day |
| Normal runtime | About 40 minutes, done by 02:40 |
| Normal volume | 1.0 to 1.4 million orders a day |
| Freshness promise | Yesterday's rows present by 06:00 UTC |
| Consumer | Finance dashboard, opened at 08:00 UTC |
Which change to orders.py lets a stranger recover Monday's revenue from that incident without making anything worse?
Once orders_daily runs under an orchestrator, every run can leave evidence behind. That evidence comes in 3 kinds called signals. Each kind answers a different question, and reaching for the wrong one is how a team ends up scrolling through thousands of log lines for an answer a graph would have shown at a glance.
Logs: what happened, in detail
A log is a timestamped record of a single event, written by the code as it runs: a task started, a retry began, a connection was refused. Logs carry the detail an investigation needs: host, attempt number, exact error. Written as named fields, they can be searched, so level = ERROR AND task = extract_orders returns every extract failure at once.
Metrics: how much, over time
| run_date | rows_loaded | duration_minutes |
|---|---|---|
| 2026-03-04 | 1,198,402 | 41 |
| 2026-03-05 | 1,214,977 | 39 |
| 2026-03-06 | 1,302,115 | 43 |
| 2026-03-07 | 1,087,560 | 38 |
| 2026-03-08 | 1,121,833 | 40 |
| 2026-03-09 | 1,233,090 | 42 |
| 2026-03-10 | 312,880 | 12 |
The March 10 row stands out: about a quarter of the usual orders, loaded in under a third of the usual time. The metric cannot say why. The logs for that run can. Among them is a warning from extract_orders reading replica lag 18h04m: the replica the job reads from had stopped copying new orders at 07:56 on March 10.
Traces: where the time went
| Span | Start | End | Minutes | Usual minutes |
|---|---|---|---|---|
| `extract_orders` | 02:00 | 02:24 | 24 | 24 |
| `transform_orders` | 02:24 | 04:49 | 145 | 10 |
| `load_revenue` | 04:49 | 04:55 | 6 | 6 |
The duration metric for that night reads 175 and says only that the run was slow. The trace puts all 135 extra minutes inside transform_orders, so the search starts in that task's logs and skips the other 2.
Finance says revenue for the last 2 weeks of March looks low next to the first 2. Which signal do you open first?
Matching the signal to the question
| Question | Signal | Why that one |
|---|---|---|
| Has the daily order count dropped this week? | Metric | A trend across many runs |
| Why did `extract_orders` fail at 02:14? | Log | The error text belongs to a single event |
| Which task stretched the March 13 run to 175 minutes? | Trace | Spans show each step's share of the time |
| Which order IDs did the transform reject? | Log | Per-record detail lives in events |
load_revenue failed on its last attempt in the run for March 16. You know the task and the run. Which signal tells you whether a rerun is worth trying?
3 monitors to ship with
- Run failureReads run status from the orchestratorFails when a task ends as failed with its retries used up. Fires earliest and names the task, but sees only runs that started.
- FreshnessReads the newest date in daily_revenueFails when yesterday's rows are missing at 06:00 UTC. Watches the output, so it notices runs that never started or never ended.
- VolumeReads the order count for the run's dateFails when the count falls outside 1.0 to 1.4 million. Judges the data, so it notices runs that succeed with the wrong amount.
orders_daily with its day-one monitors. Orders move left to right; check_volume must pass before load_revenue publishes the day, and the dashboard reads daily_revenue at 08:00 UTC.
For the volume check to act as a gate, transform_orders now writes to a staging table, stg_daily_revenue, with the same columns as daily_revenue. A fourth task, check_volume, counts the orders in that table. load_revenue copies the day across only when the count passes.
What each monitor catches alone
Each column below imagines a single monitor as the only one installed. Each row is a real way orders_daily breaks.
| What broke | Run failure alone | Freshness alone | Volume alone |
|---|---|---|---|
| `extract_orders` fails after 3 retries | Fires at 02:15 | Fires at 06:00 | Silent: check never runs |
| A deploy leaves the schedule paused | Silent: no run, no failure | Fires at 06:00 | Silent: check never runs |
| `transform_orders` hangs for 6 hours | Silent: still running | Fires at 06:00 | Silent: check never runs |
| Replica lag: 312,880 orders, run succeeds | Silent: run succeeded | Silent: the date is present | Fires before publish |
A Friday deploy leaves the orders_daily schedule paused. The team installed only the run-failure and volume monitors. When does anyone find out?
Writing the data checks
The freshness query runs at 06:00 UTC against the table above and fails when days_behind is greater than 1.
The volume query runs against the staging table after transform_orders. The orchestrator writes the run's date into it, so a rerun for March 10 counts March 10.
- ▸Duration: flag runs over 80 minutes, twice the usual 40, before they threaten 06:00
- ▸Null and uniqueness checks on
revenue_dateandregion - ▸Volume per region, once each region's normal range is known
A volume check is written as below. On March 12, the owner reruns the pipeline for March 10. Which day's orders does this check count?
3 severities
| Severity | What happens | Use it when | In orders_daily |
|---|---|---|---|
| Page | Rings the on-call engineer at any hour until acknowledged | A consumer will be harmed soon and only a person can stop it | Freshness fails at 06:00 |
| Ticket | Opens a work item handled in working hours | A person must act, but a day's wait harms no one | 3 slow nights in a row |
| Record | Writes to a dashboard or channel, interrupting nobody | Context for later, or a problem already handled | A retry that succeeded |
Page on what the consumer feels
| Monitor | Fires | Goes to |
|---|---|---|
| Run failure | When a task runs out of retries | Team channel and a ticket |
| Volume gate | Before publish | Team channel and a ticket; the load is held |
| Freshness | 06:00 UTC | A page to the on-call engineer |
| Duration over 80 minutes | After the third slow night in a row | A ticket |
| Retry that succeeded | Any time | A record only |
A different pipeline feeds a report that opens at 08:00. A rerun takes 3 hours. Its only page is a freshness check at 06:00. What should change?
Noise wears out the on-call
What a page should say
- Task failed: extract_orders
- No date, so the reader hunts for the run
- No impact, so the reader cannot judge urgency
- No link, so the reader starts from a blank screen
- orders_daily: no rows in daily_revenue for 2026-03-15 at 06:00 UTC
- Impact: finance dashboard shows stale revenue from 08:00 UTC
- Cause so far: extract_orders failed at 02:14 after 3 retries
- Runbook: runbooks/orders_daily, freshness section
- Page only when a consumer will be harmed and a person must act now
- Give every page the date and impact, plus a link to the runbook
- Demote any alert that fired twice with no action taken
- Page on a retry that went on to succeed
- Page more than once for a single incident
- Route alerts to a shared inbox nobody owns
For the third night in a row, orders_daily ran over 80 minutes. Each run still finished before 03:40. How should this alert be routed?
A runbook is the written procedure for a single alert. Its reader is the engineer the alert has reached, who may never have opened the pipeline's code. Its 5 sections run in the order that reader needs them: symptom and impact first, then diagnosis and response, with escalation last. Here is the runbook for the orders_daily freshness page.
Runbook: orders_daily freshness page
Symptom
The freshness monitor found no rows in daily_revenue for yesterday at 06:00 UTC. A normal run finishes by roughly 02:40.
Impact
Diagnosis
- Step 1. Open the orders_daily run for yesterday's date in the orchestrator. If no run exists, the schedule is paused or did not trigger: go to Response A.
- Step 2. If the run is still going, open its trace and compare each task with its usual minutes: extract 24, transform 10, load 6. Go to Response B.
- Step 3. If extract_orders, transform_orders or load_revenue failed, read the ERROR line from its last attempt. connection refused or timeout: the replica was unreachable, go to Response C. column not found or a type error: the source schema changed, go to Escalation and do not rerun.
- Step 4. If check_volume failed, run the volume query for yesterday's date and note the count. Go to Response D.
Response
- A. Unpause the schedule, then trigger a run with --date set to yesterday.
- B. If a task has run past 3 times its usual minutes, stop the run and start it again with --date. Otherwise check again at 06:30.
- C. Run SELECT 1 against orders-replica port 5432. If it answers, rerun with --date; load_revenue replaces the day's rows, so the rerun is safe. If refused, go to Escalation.
- D. Under 1.0 million usually means replica lag. Measure it on the replica with SELECT date_diff('minute', max(created_at), current_timestamp) FROM orders and rerun once it is under 10 minutes. Over 1.4 million on a known sale day: raise that date's ceiling, then rerun. Over 1.4 million otherwise: likely duplicated source rows, so page the orders database on-call and do not rerun.
- Finish every response by running the freshness query. days_behind must be 1.
Escalation
- No acknowledgement within 15 minutes: the secondary on-call is paged automatically.
- Not fixed by 07:30 UTC: tell the finance channel that yesterday's revenue is late, and when the next update will come.
- Replica down for over 30 minutes, or a schema change: page the orders database on-call.
- Owner: revenue data team. Last followed end to end on 2026-03-01.
At 06:00 on March 21 the freshness page fires. In the run for March 20, extract_orders and transform_orders succeeded. Then check_volume failed at 02:12, so load_revenue never started. The volume query returns 402,115. What does the runbook have you do?
Writing for a stranger at 3 a.m.
A draft runbook's diagnosis section reads: 'Check the logs and fix the issue.' Which rewrite helps an engineer who has never seen the pipeline?
> A marketplace pays its sellers every Friday at 09:00 UTC from a nightly seller_payouts pipeline. On a Thursday night the source replica falls behind, and the run reads 40% of the usual orders while every task reports success.
Turn a script that works into a pipeline a stranger can run, with signals, monitors, alerts and a runbook.
- Category
- Pipeline Architecture
- Difficulty
- beginner
- Duration
- 25 minutes
- Challenges
- 10 hands-on challenges
Topics covered: Script vs Operable Pipeline, Logs, Metrics, and Traces, Day-One Monitoring, Alerting That Stays Useful, A First Runbook
Lesson Sections
- Script vs Operable Pipeline
Most pipelines begin as a script written by a single engineer and understood by the same engineer. This one reads yesterday's orders, sums revenue by region, writes the totals to a warehouse table and prints a line at the end. On its author's server it has worked every night for months. The first night it fails, the people who notice are the ones reading the revenue numbers. Operable means a stranger can run it An operable pipeline can be run as a service by people who did not write it. Without
- Logs, Metrics, and Traces
Logs: what happened, in detail Metrics: how much, over time A metric is a named number recorded on every run or every minute: rows loaded, minutes taken, runs failed. Each value is a few bytes, so years of history stay cheap and draw as a line on a graph. Metrics are where you notice that something changed. They hold none of the detail that explains the change. Traces: where the time went A trace follows a single unit of work through each step it passes and records every step as a span with a st
- Day-One Monitoring
Signals sit unread until something checks them. A monitor is a scheduled check that reads a signal and returns pass or fail. A new pipeline needs only the handful of monitors that between them catch its most common breakages, and each one should catch something the others cannot. 3 monitors to ship with What each monitor catches alone Freshness fires in 3 of the 4 rows, which makes it the monitor to keep if only a single one were allowed: it watches what the consumer will see, so it notices fail
- Alerting That Stays Useful
A monitor decides pass or fail. An alert decides who hears about a failure, and how fast. Keeping the 2 decisions apart lets every monitor record its result while only a few results interrupt a person. 3 severities Choosing between a page and a ticket comes down to 2 durations: how long until a consumer is harmed, and how long a fix takes. At 06:00 the dashboard is 2 hours away and a rerun takes about 40 minutes, so a 06:00 page leaves room to finish the repair. Page on what the consumer feels P
- A First Runbook
Runbook: orders_daily freshness page Symptom Impact Finance opens the revenue dashboard at 08:00 UTC. Until the load lands, the newest day it shows is the day before yesterday. No other team reads this table. Diagnosis Response Escalation Writing for a stranger at 3 a.m.