The Life of a Task Instance
Concepts covered: Task instance states
A DAG file names tasks. A task instance is the thing Airflow runs: one task, inside one run, for one window of time. The nightly_orders DAG in this section has 3 tasks, extract, transform and load, so every run it creates holds 3 task instances, and the grid draws each one as a cell, the run down the column and the task along the row. Every state in this lesson belongs to a cell. The DAG file itself has no state.
3 names pin a cell down: the dag_id, the task_id and the run_id, which for a scheduled run is the word scheduled, 2 underscores and the window's logical date, such as scheduled__2026-03-09T00:00:00+00:00. A fourth number rides along: the try number, which counts how many times a worker has started this cell's code. The first start is try 1, and the number only ever grows.
6 states on the way to done
| State | Who writes it | What it means |
|---|---|---|
| none | the scheduler, when the run is created | The cell exists and nothing has looked at it yet. The grid draws it blank. |
| scheduled | the scheduler | Every task this one depends on ended in success, so it may run as soon as a worker is free. |
| queued | the scheduler, handing it off | It is in the line for a worker. None of the task's code has run. |
| running | the worker | The task's code is executing right now, in whatever try this is. |
| success | the worker, reporting back | The code returned without raising. Final. |
| failed | the worker, reporting back | The code raised and no tries are left. Final. |
2 parties move a cell. The scheduler owns the top half of the table: it creates the cell as none when the run appears, moves it to scheduled once the tasks above it are done, and to queued when it hands the task to the worker pool. A worker owns the bottom half: it takes the task from the queue, writes running, executes the Python, and reports success or failed when the code returns or raises. The scheduler decides when a cell may run; the worker decides how it ends. Every later state in this lesson is one of these 2 parties telling the other something about a cell.
from datetime import datetime
from airflow.sdk import dag, task
@dag(
schedule="@daily",
start_date=datetime(2026, 3, 1),
catchup=False,
)
def nightly_orders():
@task
def extract(): ...
@task
def transform(): ...
@task
def load(): ...
extract() >> transform() >> load()
nightly_orders()
The order of events, cell by cell
| Event | extract | transform | load |
|---|---|---|---|
| Window closes, run appears | none | none | none |
| Scheduler looks at the run | scheduled | none | none |
| extract handed to the worker pool | queued | none | none |
| A worker picks it up | running | none | none |
| extract returns | success | scheduled | none |
| transform handed off, picked up | success | running | none |
| transform returns | success | success | scheduled |
| load picked up, returns | success | success | success |
Put clock times on those rows. The window for 2026-03-09 closes at midnight, so the run appears at 00:00. extract spends a few seconds in scheduled and queued while the scheduler hands it off and a worker picks it up, then runs for 2 minutes. transform starts near 00:02 and ends near 00:05, and load ends near 00:07. The start and end times you read when you pause on a cell cover running only: a cell that waited 4 minutes in queued before a 2 minute run shows a 2 minute duration and a start time 4 minutes after its run appeared.
success and failed are final states. The scheduler never moves a final cell on its own, and the run takes its own state from its cells: a run stays running while any cell is not final, ends success when every cell is final and none of them is failed, and ends failed otherwise. The rest of this lesson is about the 3 things that bend the straight line in the table above: a task that gets another try, a failure that spreads to the cells below it, and a person clearing a final cell so the scheduler sees it as new.
A cell that sits in queued for minutes is waiting on a free worker; a later lesson on executors and capacity covers why that happens and how to size for it.
About this lesson
This section is part of the Task Lifecycle: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.