Task vs DAG vs Run for Retries
When something breaks at 04:00, people use 5 words loosely, and the right fix depends on which one you mean. Take a DAG named daily_revenue with 3 tasks in a line: extract_bookings, then convert_currency, then publish_report.
| Term | What it is | In daily_revenue | How many |
|---|---|---|---|
| DAG | the definition: tasks, edges, schedule | `daily_revenue` as written in code | 1 |
| Task | a single node in the definition | `convert_currency` | 3 |
| DAG run | a single execution of the DAG for a single data interval | the run for 2026-10-23 | 1 per interval |
| Task instance | a task inside a specific run | `convert_currency` in the 2026-10-23 run | 3 per run |
| Attempt | a single try at a task instance | try 2 of that task instance | the retry count plus 1 |
Daily runs over 30 days produce 30 DAG runs and 90 task instances from 1 DAG and 3 tasks. Every state the orchestrator shows (queued, running, success, failed, up for retry) belongs to a task instance or a run. Each run is identified by its data interval, which the schedule cuts: at 03:00, the 2026-10-23 run covers 03:00 on the 23rd to 03:00 on the 24th and starts as it closes. A task needing calendar 23 October filters on the run's date.
- dag: dag_id varchar PK, schedule varchar, owner varchar
- dag_run: run_id varchar PK, dag_id varchar FK, data_interval_start timestamp, data_interval_end timestamp, state varchar
- task_instance: task_instance_id bigint PK, run_id varchar FK, task_id varchar, try_number integer, state varchar
- dag_run.dag_id references dag.dag_id (N:1)
- task_instance.run_id references dag_run.run_id (N:1)
Success and failure live on task_instance rows, so every fix below acts on runs and task instances.
Automatic retries act on a single task instance
Say convert_currency fails in the 2026-10-23 run with a retry count of 2. The orchestrator leaves the rest of the run alone: it moves that task instance to up for retry, waits the configured delay, and starts attempt 2. extract_bookings keeps its success and doesn't execute again, and publish_report keeps waiting. A retry count of 2 allows 3 attempts. Only when attempt 3 fails does the task instance become failed, and only then does the run fail.
dag_id: daily_revenue
schedule: "0 3 * * *"
timezone: UTC
default_retries: 2
retry_delay_minutes: 5
tasks:
- id: extract_bookings
retries: 4
- id: convert_currency
depends_on: [extract_bookings]
- id: publish_report
depends_on: [convert_currency]
retries: 0An hourly DAG has 4 tasks, each with a retry count of 1. After 7 days, how many task instances exist, and how many attempts did a task instance that ended failed use?
- Task instances: 672; attempts on the failed one: 2
- Task instances: 168; attempts on the failed one: 2
- Task instances: 672; attempts on the failed one: 3
- Task instances: 4; attempts on the failed one: 2
Choosing the scope of a manual rerun
Automatic retries suit a task that may succeed a few minutes later on the same inputs. When a person steps in after fixing something, there are 4 scopes, and each re-executes a different set of task instances.
| Action | Re-executes | Use it when |
|---|---|---|
| Clear a single task instance | that task instance only | it failed and nothing downstream has run yet |
| Clear with downstream | that task instance and every later one in the same run | its output changed after later tasks read it |
| Clear the whole run | every task instance in the run | the run's inputs were wrong from the first task |
| Trigger a new run | a fresh run with a new run id | you want an extra execution that leaves existing runs untouched |
The common mistake is clearing too little. Suppose extract_bookings for 2026-10-23 succeeded while its source was still filling in. You wait for the source to finish and clear only extract_bookings. It reruns and succeeds, while convert_currency and publish_report keep their earlier success, so the report still shows partial numbers. Once a task's output changes, every task that read it must run again in the same run, which is what clearing with downstream does.
The other mistake is triggering a new run to repair an old one. A manual run gets its own run id and, depending on how it is started, its own data interval. A report task that filters on its run's interval then builds numbers for 24 October and leaves 23 October broken. To repair an interval, act on that interval's run.
In the 2026-10-23 run of daily_revenue, all 3 tasks succeeded. Then you learn the rates table that convert_currency read was missing 2 currencies, and the table is now fixed. What should you do?
- Clear convert_currency only, so the conversion reruns on fixed rates
- Trigger a new run of daily_revenue so every task runs again
- Clear convert_currency with downstream in the 2026-10-23 run
- Clear the whole 2026-10-23 run, starting from extract_bookings
About this lesson
This section is part of the Orchestration: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.