BeginnerPipeline Architecture · 25 min · 10 challenges

Pipeline Operations: Beginner

An engineer is 3 weeks into a Google SRE on-call rotation. At 06:00 UTC her pager fires: yesterday's rows are missing from a revenue rollup she has never opened, and finance reads it at 08:00. Its author is asleep. Whether those 2 hours are enough depends on choices made months earlier: which signals the job emits, which alert paged her, and whether a runbook says what to try first.

Script vs Operable Pipeline

Daily Life
Interviews
Most pipelines begin as a script written by a single engineer and understood by the same engineer. This one reads yesterday's orders, sums revenue by region, writes the totals to a warehouse table and prints a line at the end. On its author's server it has worked every night for months. The first night it fails, the people who notice are the ones reading the revenue numbers.
import datetime
import db
import warehouse
def main():
day = datetime.date.today() - datetime.timedelta(days=1)
rows = db.query("SELECT region, amount FROM orders WHERE order_date = %s", day)
totals = {}
for row in rows:
totals[row["region"]] = totals.get(row["region"], 0) + row["amount"]
warehouse.insert("daily_revenue", day, totals)
print("done")
main()
What the script leaves unanswered
  • print("done") writes to a terminal nobody watches. Seen from anywhere else, a failed night and a good night look the same.
  • date.today() ties the processed day to the clock, so once Monday has passed there is no way to ask for Monday again.
  • warehouse.insert appends. Two runs for the same day write that day's totals twice, the duplicate-row bug from Idempotent Reruns.
  • Nothing records the row count or the run time, so nobody can say whether a night's 312,880 orders is normal or a warning.
Multiple Choice

orders.py fails at 02:00 on Tuesday. The Wednesday 02:00 run succeeds. At 09:00 on Wednesday an analyst spots a gap and runs the script once more by hand. What does daily_revenue now hold for Monday and Tuesday?

Operable means a stranger can run it

An operable pipeline can be run as a service by people who did not write it. Without opening the code, a stranger can find out whether last night's run happened, whether its output can be trusted, who must hear when it cannot, and what to do next. Each of those 4 answers is work the script never did.
✓Working script
  • Started by cron on a single server
  • Processes whatever day the clock says
  • Appends on every run
  • Reports success to a terminal
  • Breakage found by the people reading dashboards
✗Operable pipeline
  • Scheduled and retried by an orchestrator
  • Takes the date to process as an input
  • Replaces the day's rows, so reruns are safe
  • Sends its logs and metrics somewhere searchable, along with the status of each run
  • Monitors watch the output and alerts reach an owner, with a runbook to follow
The 3 a.m. test
Suppose the pipeline breaks at 03:00 and the only person awake has never looked at it. Can they tell it is broken before a customer says so, and how much it matters? Can they fix it, or hand it to the right team, without phoning the author? Every 'no' marks a missing piece.

The rest of this lesson works on orders_daily, the same job rebuilt as a DAG under an orchestrator. Its facts stay fixed in every section:

PropertyValue
Pipeline`orders_daily`
ScheduleDaily at 02:00 UTC, processing the previous day
Tasks`extract_orders`, `transform_orders`, `load_revenue`
SourceOrders database, read through a replica
Output`daily_revenue`, a row per region per day
Normal runtimeAbout 40 minutes, done by 02:40
Normal volume1.0 to 1.4 million orders a day
Freshness promiseYesterday's rows present by 06:00 UTC
ConsumerFinance dashboard, opened at 08:00 UTC
TIP
Name an owning team before the pipeline ships. An alert from a pipeline that nobody owns reaches nobody, however well it is written.
Multiple Choice

Which change to orders.py lets a stranger recover Monday's revenue from that incident without making anything worse?

Logs, Metrics, and Traces

Daily Life
Interviews

Once orders_daily runs under an orchestrator, every run can leave evidence behind. That evidence comes in 3 kinds called signals. Each kind answers a different question, and reaching for the wrong one is how a team ends up scrolling through thousands of log lines for an answer a graph would have shown at a glance.

Logs: what happened, in detail

A log is a timestamped record of a single event, written by the code as it runs: a task started, a retry began, a connection was refused. Logs carry the detail an investigation needs: host, attempt number, exact error. Written as named fields, they can be searched, so level = ERROR AND task = extract_orders returns every extract failure at once.

{
"ts": "2026-03-16T02:14:07Z",
"level": "ERROR",
"pipeline": "orders_daily",
"task": "extract_orders",
"run_date": "2026-03-15",
"attempt": 4,
"message": "connection refused",
"host": "orders-replica",
"port": 5432
}

Metrics: how much, over time

A metric is a named number recorded on every run or every minute: rows loaded, minutes taken, runs failed. Each value is a few bytes, so years of history stay cheap and draw as a line on a graph. Metrics are where you notice that something changed. They hold none of the detail that explains the change.
run_daterows_loadedduration_minutes
2026-03-041,198,40241
2026-03-051,214,97739
2026-03-061,302,11543
2026-03-071,087,56038
2026-03-081,121,83340
2026-03-091,233,09042
2026-03-10312,88012

The March 10 row stands out: about a quarter of the usual orders, loaded in under a third of the usual time. The metric cannot say why. The logs for that run can. Among them is a warning from extract_orders reading replica lag 18h04m: the replica the job reads from had stopped copying new orders at 07:56 on March 10.

Traces: where the time went

A trace follows a single unit of work through each step it passes and records every step as a span with a start and an end. In a batch pipeline, a run is the trace and each task is a span. Traces answer the question the other 2 signals leave open: which step made a slow run slow? This is the trace for March 13, a run that ended at 04:55.
SpanStartEndMinutesUsual minutes
`extract_orders`02:0002:242424
`transform_orders`02:2404:4914510
`load_revenue`04:4904:5566

The duration metric for that night reads 175 and says only that the run was slow. The trace puts all 135 extra minutes inside transform_orders, so the search starts in that task's logs and skips the other 2.

Multiple Choice

Finance says revenue for the last 2 weeks of March looks low next to the first 2. Which signal do you open first?

Matching the signal to the question

QuestionSignalWhy that one
Has the daily order count dropped this week?MetricA trend across many runs
Why did `extract_orders` fail at 02:14?LogThe error text belongs to a single event
Which task stretched the March 13 run to 175 minutes?TraceSpans show each step's share of the time
Which order IDs did the transform reject?LogPer-record detail lives in events
TIP
Investigations often use the signals in turn: a metric shows that something changed, a trace narrows it to a step, and that step's logs explain why.
Multiple Choice

load_revenue failed on its last attempt in the run for March 16. You know the task and the run. Which signal tells you whether a rerun is worth trying?

Day-One Monitoring

Daily Life
Interviews
Signals sit unread until something checks them. A monitor is a scheduled check that reads a signal and returns pass or fail. A new pipeline needs only the handful of monitors that between them catch its most common breakages, and each one should catch something the others cannot.

3 monitors to ship with

  • Run failureReads run status from the orchestratorFails when a task ends as failed with its retries used up. Fires earliest and names the task, but sees only runs that started.
  • FreshnessReads the newest date in daily_revenueFails when yesterday's rows are missing at 06:00 UTC. Watches the output, so it notices runs that never started or never ended.
  • VolumeReads the order count for the run's dateFails when the count falls outside 1.0 to 1.4 million. Judges the data, so it notices runs that succeed with the wrong amount.
operational database
orders replica
orchestrator
extract orders
RETRY3MONITORrun failure
orchestrator
transform orders
RETRY3MONITORrun failure
staging table
stg daily revenue
orchestrator
check volume
ERRORhold the load outside 1.0M to 1.4M
orchestrator
load revenue
RETRY3IDEMPOTENCYreplace the day's rowsMONITORrun failure
warehouse table
daily revenue
SLAyesterday by 06:00 UTC
dashboard
finance dashboard
08:00 UTC

orders_daily with its day-one monitors. Orders move left to right; check_volume must pass before load_revenue publishes the day, and the dashboard reads daily_revenue at 08:00 UTC.

For the volume check to act as a gate, transform_orders now writes to a staging table, stg_daily_revenue, with the same columns as daily_revenue. A fourth task, check_volume, counts the orders in that table. load_revenue copies the day across only when the count passes.

What each monitor catches alone

Each column below imagines a single monitor as the only one installed. Each row is a real way orders_daily breaks.

What brokeRun failure aloneFreshness aloneVolume alone
`extract_orders` fails after 3 retriesFires at 02:15Fires at 06:00Silent: check never runs
A deploy leaves the schedule pausedSilent: no run, no failureFires at 06:00Silent: check never runs
`transform_orders` hangs for 6 hoursSilent: still runningFires at 06:00Silent: check never runs
Replica lag: 312,880 orders, run succeedsSilent: run succeededSilent: the date is presentFires before publish
Freshness fires in 3 of the 4 rows, which makes it the monitor to keep if only a single one were allowed: it watches what the consumer will see, so it notices failures no task reported. Run failure fires 3 hours and 45 minutes sooner in the first row and names the broken task. Volume is the only check that judges the data; without it, the quarter-sized March 10 load reaches the dashboard.
Multiple Choice

A Friday deploy leaves the orders_daily schedule paused. The team installed only the run-failure and volume monitors. When does anyone find out?

Writing the data checks

daily_revenue
revenue_datePKDATE
regionPKVARCHAR
revenueDECIMAL(18,2)
order_countBIGINT
loaded_atTIMESTAMP

The freshness query runs at 06:00 UTC against the table above and fails when days_behind is greater than 1.

SELECT
MAX(revenue_date) AS latest_date,
DATE_DIFF('day', MAX(revenue_date), CURRENT_DATE) AS days_behind
FROM daily_revenue

The volume query runs against the staging table after transform_orders. The orchestrator writes the run's date into it, so a rerun for March 10 counts March 10.

SELECT
SUM(order_count) AS orders
FROM stg_daily_revenue
WHERE revenue_date = DATE '2026-03-10'
TIP
Only the freshness query reads the clock, because it asks about the present: is yesterday loaded yet? Every check that judges a run's data should use that run's date.
Set the range wide, from a few weeks of history; 1.0 to 1.4 million covers every normal March night. A 1.9-million-order sale day will trip it on good data, and the owner raises the ceiling for that date.
What can wait until week 2
  • ▸Duration: flag runs over 80 minutes, twice the usual 40, before they threaten 06:00
  • ▸Null and uniqueness checks on revenue_date and region
  • ▸Volume per region, once each region's normal range is known
Multiple Choice

A volume check is written as below. On March 12, the owner reruns the pipeline for March 10. Which day's orders does this check count?

SELECT
SUM(order_count) AS orders
FROM stg_daily_revenue
WHERE revenue_date = CURRENT_DATE - INTERVAL '1' DAY

Alerting That Stays Useful

Daily Life
Interviews
A monitor decides pass or fail. An alert decides who hears about a failure, and how fast. Keeping the 2 decisions apart lets every monitor record its result while only a few results interrupt a person.

3 severities

SeverityWhat happensUse it whenIn orders_daily
PageRings the on-call engineer at any hour until acknowledgedA consumer will be harmed soon and only a person can stop itFreshness fails at 06:00
TicketOpens a work item handled in working hoursA person must act, but a day's wait harms no one3 slow nights in a row
RecordWrites to a dashboard or channel, interrupting nobodyContext for later, or a problem already handledA retry that succeeded
Choosing between a page and a ticket comes down to 2 durations: how long until a consumer is harmed, and how long a fix takes. At 06:00 the dashboard is 2 hours away and a rerun takes about 40 minutes, so a 06:00 page leaves room to finish the repair.

Page on what the consumer feels

Pages work best on symptoms, the effects a consumer would notice: data that is late or missing, or numbers that are wrong. Causes such as a refused connection or a slow task fit tickets and records better, since retries absorb many of them. A symptom page also stands in for the several cause alerts a single incident sets off.
MonitorFiresGoes to
Run failureWhen a task runs out of retriesTeam channel and a ticket
Volume gateBefore publishTeam channel and a ticket; the load is held
Freshness06:00 UTCA page to the on-call engineer
Duration over 80 minutesAfter the third slow night in a rowA ticket
Retry that succeededAny timeA record only
With this routing, the extract outage in the run for March 15 produces a single page. The run-failure post from 02:15 is already in the channel when the freshness page arrives at 06:00, so the engineer starts with the cause named.
Multiple Choice

A different pipeline feeds a report that opens at 08:00. A rerun takes 3 hours. Its only page is a freshness check at 06:00. What should change?

Noise wears out the on-call

Each page that turns out to need no action teaches the on-call engineer that pages can wait. After enough of them, a real page waits too. Picture a team whose alert history for a week looked like this:
41
pages in a single week
3
pages that needed a person
Most of the rest were retries that later succeeded and volume ranges drawn too tight. Demoting the first to records and widening the second removes the noise and hides no real incident.

What a page should say

✓Hard to act on
  • Task failed: extract_orders
  • No date, so the reader hunts for the run
  • No impact, so the reader cannot judge urgency
  • No link, so the reader starts from a blank screen
✗Ready to act on
  • orders_daily: no rows in daily_revenue for 2026-03-15 at 06:00 UTC
  • Impact: finance dashboard shows stale revenue from 08:00 UTC
  • Cause so far: extract_orders failed at 02:14 after 3 retries
  • Runbook: runbooks/orders_daily, freshness section
✓Do
  • Page only when a consumer will be harmed and a person must act now
  • Give every page the date and impact, plus a link to the runbook
  • Demote any alert that fired twice with no action taken
✗Don't
  • Page on a retry that went on to succeed
  • Page more than once for a single incident
  • Route alerts to a shared inbox nobody owns
Multiple Choice

For the third night in a row, orders_daily ran over 80 minutes. Each run still finished before 03:40. How should this alert be routed?

A First Runbook

Daily Life
Interviews

A runbook is the written procedure for a single alert. Its reader is the engineer the alert has reached, who may never have opened the pipeline's code. Its 5 sections run in the order that reader needs them: symptom and impact first, then diagnosis and response, with escalation last. Here is the runbook for the orders_daily freshness page.

Runbook: orders_daily freshness page

//

Symptom

The freshness monitor found no rows in daily_revenue for yesterday at 06:00 UTC. A normal run finishes by roughly 02:40.

//

Impact

Finance opens the revenue dashboard at 08:00 UTC. Until the load lands, the newest day it shows is the day before yesterday. No other team reads this table.
//

Diagnosis

  • Step 1. Open the orders_daily run for yesterday's date in the orchestrator. If no run exists, the schedule is paused or did not trigger: go to Response A.
  • Step 2. If the run is still going, open its trace and compare each task with its usual minutes: extract 24, transform 10, load 6. Go to Response B.
  • Step 3. If extract_orders, transform_orders or load_revenue failed, read the ERROR line from its last attempt. connection refused or timeout: the replica was unreachable, go to Response C. column not found or a type error: the source schema changed, go to Escalation and do not rerun.
  • Step 4. If check_volume failed, run the volume query for yesterday's date and note the count. Go to Response D.
//

Response

  • A. Unpause the schedule, then trigger a run with --date set to yesterday.
  • B. If a task has run past 3 times its usual minutes, stop the run and start it again with --date. Otherwise check again at 06:30.
  • C. Run SELECT 1 against orders-replica port 5432. If it answers, rerun with --date; load_revenue replaces the day's rows, so the rerun is safe. If refused, go to Escalation.
  • D. Under 1.0 million usually means replica lag. Measure it on the replica with SELECT date_diff('minute', max(created_at), current_timestamp) FROM orders and rerun once it is under 10 minutes. Over 1.4 million on a known sale day: raise that date's ceiling, then rerun. Over 1.4 million otherwise: likely duplicated source rows, so page the orders database on-call and do not rerun.
  • Finish every response by running the freshness query. days_behind must be 1.
//

Escalation

  • No acknowledgement within 15 minutes: the secondary on-call is paged automatically.
  • Not fixed by 07:30 UTC: tell the finance channel that yesterday's revenue is late, and when the next update will come.
  • Replica down for over 30 minutes, or a schema change: page the orders database on-call.
  • Owner: revenue data team. Last followed end to end on 2026-03-01.
Multiple Choice

At 06:00 on March 21 the freshness page fires. In the run for March 20, extract_orders and transform_orders succeeded. Then check_volume failed at 02:12, so load_revenue never started. The volume query returns 402,115. What does the runbook have you do?

Writing for a stranger at 3 a.m.

Name the exact query, command or screen; 'check the logs' names none of them
Name the exact query, command or screen; 'check the logs' names none of them
Give each check its possible results and the next step for every result
Give each check its possible results and the next step for every result
Say when a rerun would make things worse, as well as when it helps
Say when a rerun would make things worse, as well as when it helps
Have an engineer new to the pipeline follow it in working hours, and record the date
Have an engineer new to the pipeline follow it in working hours, and record the date
Update it after each incident, while the missing steps are fresh
Update it after each incident, while the missing steps are fresh
Multiple Choice

A draft runbook's diagnosis section reads: 'Check the logs and fix the issue.' Which rewrite helps an engineer who has never seen the pipeline?

❯❯❯PUTTING IT ALL TOGETHER

> A marketplace pays its sellers every Friday at 09:00 UTC from a nightly seller_payouts pipeline. On a Thursday night the source replica falls behind, and the run reads 40% of the usual orders while every task reports success.

The volume check, run against Thursday's date, holds the load, so no seller is paid from partial numbers.
The run-failure post names the gate, and the order-count metric shows the drop happened on a single night.
Freshness pages at its deadline because sellers are harmed within hours and a rerun fits inside that window.
The runbook's volume step sends the on-call engineer to check replica lag and wait for it to clear before rerunning; replacing the day's rows makes the rerun safe.
If the replica stays down, the escalation section tells the payments team before 09:00 that payouts will be late.
KEY TAKEAWAYS
A script becomes an operable pipeline when a stranger can tell it ran, trust its output, know who to tell and know what to do, all without reading the code.
Metrics show that something changed, traces show which step changed, and logs show why.
Ship with 3 monitors: run failure for speed and detail, freshness for anything that keeps data from arriving, volume for runs that succeed with the wrong data.
Checks on a run's data use the run's date; only the freshness check reads the clock.
Page when a consumer will be harmed and only a person can stop it in time; send everything else to tickets and records.
A runbook gives symptom, impact, ordered diagnosis checks, a response for each cause, and who to escalate to and when.

Turn a script that works into a pipeline a stranger can run, with signals, monitors, alerts and a runbook.

Category
Pipeline Architecture
Difficulty
beginner
Duration
25 minutes
Challenges
10 hands-on challenges

Topics covered: Script vs Operable Pipeline, Logs, Metrics, and Traces, Day-One Monitoring, Alerting That Stays Useful, A First Runbook

Lesson Sections

  1. Script vs Operable Pipeline

    Most pipelines begin as a script written by a single engineer and understood by the same engineer. This one reads yesterday's orders, sums revenue by region, writes the totals to a warehouse table and prints a line at the end. On its author's server it has worked every night for months. The first night it fails, the people who notice are the ones reading the revenue numbers. Operable means a stranger can run it An operable pipeline can be run as a service by people who did not write it. Without

  2. Logs, Metrics, and Traces

    Logs: what happened, in detail Metrics: how much, over time A metric is a named number recorded on every run or every minute: rows loaded, minutes taken, runs failed. Each value is a few bytes, so years of history stay cheap and draw as a line on a graph. Metrics are where you notice that something changed. They hold none of the detail that explains the change. Traces: where the time went A trace follows a single unit of work through each step it passes and records every step as a span with a st

  3. Day-One Monitoring

    Signals sit unread until something checks them. A monitor is a scheduled check that reads a signal and returns pass or fail. A new pipeline needs only the handful of monitors that between them catch its most common breakages, and each one should catch something the others cannot. 3 monitors to ship with What each monitor catches alone Freshness fires in 3 of the 4 rows, which makes it the monitor to keep if only a single one were allowed: it watches what the consumer will see, so it notices fail

  4. Alerting That Stays Useful

    A monitor decides pass or fail. An alert decides who hears about a failure, and how fast. Keeping the 2 decisions apart lets every monitor record its result while only a few results interrupt a person. 3 severities Choosing between a page and a ticket comes down to 2 durations: how long until a consumer is harmed, and how long a fix takes. At 06:00 the dashboard is 2 hours away and a rerun takes about 40 minutes, so a 06:00 page leaves room to finish the repair. Page on what the consumer feels P

  5. A First Runbook

    Runbook: orders_daily freshness page Symptom Impact Finance opens the revenue dashboard at 08:00 UTC. Until the load lands, the newest day it shows is the day before yesterday. No other team reads this table. Diagnosis Response Escalation Writing for a stranger at 3 a.m.