Logs, Metrics, and Traces

Once orders_daily runs under an orchestrator, every run can leave evidence behind. That evidence comes in 3 kinds called signals. Each kind answers a different question, and reaching for the wrong one is how a team ends up scrolling through thousands of log lines for an answer a graph would have shown at a glance.

Logs: what happened, in detail

A log is a timestamped record of a single event, written by the code as it runs: a task started, a retry began, a connection was refused. Logs carry the detail an investigation needs: host, attempt number, exact error. Written as named fields, they can be searched, so level = ERROR AND task = extract_orders returns every extract failure at once.

{
  "ts": "2026-03-16T02:14:07Z",
  "level": "ERROR",
  "pipeline": "orders_daily",
  "task": "extract_orders",
  "run_date": "2026-03-15",
  "attempt": 4,
  "message": "connection refused",
  "host": "orders-replica",
  "port": 5432
}

Metrics: how much, over time

A metric is a named number recorded on every run or every minute: rows loaded, minutes taken, runs failed. Each value is a few bytes, so years of history stay cheap and draw as a line on a graph. Metrics are where you notice that something changed. They hold none of the detail that explains the change.

run_daterows_loadedduration_minutes
2026-03-041,198,40241
2026-03-051,214,97739
2026-03-061,302,11543
2026-03-071,087,56038
2026-03-081,121,83340
2026-03-091,233,09042
2026-03-10312,88012

The March 10 row stands out: about a quarter of the usual orders, loaded in under a third of the usual time. The metric cannot say why. The logs for that run can. Among them is a warning from extract_orders reading replica lag 18h04m: the replica the job reads from had stopped copying new orders at 07:56 on March 10.

Traces: where the time went

A trace follows a single unit of work through each step it passes and records every step as a span with a start and an end. In a batch pipeline, a run is the trace and each task is a span. Traces answer the question the other 2 signals leave open: which step made a slow run slow? This is the trace for March 13, a run that ended at 04:55.

SpanStartEndMinutesUsual minutes
`extract_orders`02:0002:242424
`transform_orders`02:2404:4914510
`load_revenue`04:4904:5566

The duration metric for that night reads 175 and says only that the run was slow. The trace puts all 135 extra minutes inside transform_orders, so the search starts in that task's logs and skips the other 2.

Finance says revenue for the last 2 weeks of March looks low next to the first 2. Which signal do you open first?

Matching the signal to the question

QuestionSignalWhy that one
Has the daily order count dropped this week?MetricA trend across many runs
Why did `extract_orders` fail at 02:14?LogThe error text belongs to a single event
Which task stretched the March 13 run to 175 minutes?TraceSpans show each step's share of the time
Which order IDs did the transform reject?LogPer-record detail lives in events

load_revenue failed on its last attempt in the run for March 16. You know the task and the run. Which signal tells you whether a rerun is worth trying?

About this lesson

This section is part of the Pipeline Operations: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.