Logs, Metrics, and Traces
Once orders_daily runs under an orchestrator, every run can leave evidence behind. That evidence comes in 3 kinds called signals. Each kind answers a different question, and reaching for the wrong one is how a team ends up scrolling through thousands of log lines for an answer a graph would have shown at a glance.
Logs: what happened, in detail
A log is a timestamped record of a single event, written by the code as it runs: a task started, a retry began, a connection was refused. Logs carry the detail an investigation needs: host, attempt number, exact error. Written as named fields, they can be searched, so level = ERROR AND task = extract_orders returns every extract failure at once.
{
"ts": "2026-03-16T02:14:07Z",
"level": "ERROR",
"pipeline": "orders_daily",
"task": "extract_orders",
"run_date": "2026-03-15",
"attempt": 4,
"message": "connection refused",
"host": "orders-replica",
"port": 5432
}Metrics: how much, over time
A metric is a named number recorded on every run or every minute: rows loaded, minutes taken, runs failed. Each value is a few bytes, so years of history stay cheap and draw as a line on a graph. Metrics are where you notice that something changed. They hold none of the detail that explains the change.
| run_date | rows_loaded | duration_minutes |
|---|---|---|
| 2026-03-04 | 1,198,402 | 41 |
| 2026-03-05 | 1,214,977 | 39 |
| 2026-03-06 | 1,302,115 | 43 |
| 2026-03-07 | 1,087,560 | 38 |
| 2026-03-08 | 1,121,833 | 40 |
| 2026-03-09 | 1,233,090 | 42 |
| 2026-03-10 | 312,880 | 12 |
The March 10 row stands out: about a quarter of the usual orders, loaded in under a third of the usual time. The metric cannot say why. The logs for that run can. Among them is a warning from extract_orders reading replica lag 18h04m: the replica the job reads from had stopped copying new orders at 07:56 on March 10.
Traces: where the time went
A trace follows a single unit of work through each step it passes and records every step as a span with a start and an end. In a batch pipeline, a run is the trace and each task is a span. Traces answer the question the other 2 signals leave open: which step made a slow run slow? This is the trace for March 13, a run that ended at 04:55.
| Span | Start | End | Minutes | Usual minutes |
|---|---|---|---|---|
| `extract_orders` | 02:00 | 02:24 | 24 | 24 |
| `transform_orders` | 02:24 | 04:49 | 145 | 10 |
| `load_revenue` | 04:49 | 04:55 | 6 | 6 |
The duration metric for that night reads 175 and says only that the run was slow. The trace puts all 135 extra minutes inside transform_orders, so the search starts in that task's logs and skips the other 2.
Finance says revenue for the last 2 weeks of March looks low next to the first 2. Which signal do you open first?
- The logs for every run in March, filtered to lines at the ERROR level.
- The trace for the most recent run, to see which task took the most time.
- The rows_loaded metric for March, to see when the daily count changed and by how much.
- The orchestrator's list of failed runs, since low revenue means some runs must have failed.
Matching the signal to the question
| Question | Signal | Why that one |
|---|---|---|
| Has the daily order count dropped this week? | Metric | A trend across many runs |
| Why did `extract_orders` fail at 02:14? | Log | The error text belongs to a single event |
| Which task stretched the March 13 run to 175 minutes? | Trace | Spans show each step's share of the time |
| Which order IDs did the transform reject? | Log | Per-record detail lives in events |
load_revenue failed on its last attempt in the run for March 16. You know the task and the run. Which signal tells you whether a rerun is worth trying?
- Its log, because the error text shows whether the cause is transient, like a refused connection, or permanent, like a missing column.
- Its duration metric, because a long attempt shows the task got far enough that a rerun will finish.
- Its trace, because span lengths show which task failed, and that is enough to decide on a rerun.
- The failed-runs metric, because a high count this month marks the error as a known pattern to rerun.
About this lesson
This section is part of the Pipeline Operations: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.