Task Lifecycle: Beginner

A DAG file names tasks. A task instance is the thing Airflow runs: one task, inside one run, for one window of time. The nightly_orders DAG in this section has 3 tasks, extract, transform and load, so every run it creates holds 3 task instances, and the grid draws each one as a cell, the run down the column and the task along the row. Every state in this lesson belongs to a cell. The DAG file itself has no state.

3 names pin a cell down: the dag_id, the task_id and the run_id, which for a scheduled run is the word scheduled, 2 underscores and the window's logical date, such as scheduled__2026-03-09T00:00:00+00:00. A fourth number rides along: the try number, which counts how many times a worker has started this cell's code. The first start is try 1, and the number only ever grows.

2 parties move a cell. The scheduler owns the top half of the table: it creates the cell as none when the run appears, moves it to scheduled once the tasks above it are done, and to queued when it hands the task to the worker pool. A worker owns the bottom half: it takes the task from the queue, writes running, executes the Python, and reports success or failed when the code returns or raises. The scheduler decides when a cell may run; the worker decides how it ends. Every later state in this lesson is one of these 2 parties telling the other something about a cell.

Put clock times on those rows. The window for 2026-03-09 closes at midnight, so the run appears at 00:00. extract spends a few seconds in scheduled and queued while the scheduler hands it off and a worker picks it up, then runs for 2 minutes. transform starts near 00:02 and ends near 00:05, and load ends near 00:07. The start and end times you read when you pause on a cell cover running only: a cell that waited 4 minutes in queued before a 2 minute run shows a 2 minute duration and a start time 4 minutes after its run appeared.

success and failed are final states. The scheduler never moves a final cell on its own, and the run takes its own state from its cells: a run stays running while any cell is not final, ends success when every cell is final and none of them is failed, and ends failed otherwise. The rest of this lesson is about the 3 things that bend the straight line in the table above: a task that gets another try, a failure that spreads to the cells below it, and a person clearing a final cell so the scheduler sees it as new.

A cell that sits in queued for minutes is waiting on a free worker; a later lesson on executors and capacity covers why that happens and how to size for it.

Picture the nightly_orders run at 00:08. extract took 2 minutes and transform took 3, both ended success, and load has been running for 3 minutes when the warehouse drops its connection and the task raises. The 2 cells above it did their work, and the failure says nothing about their output. If the unit of a retry were the whole run, the fix would redo 5 minutes of finished work to reach the 1 task that broke. So Airflow retries the task instance alone: the 2 green cells stay green, and only load gets another try.

2 settings on a task control this. retries is how many extra tries the task may take after the first one raises, and it defaults to 0. retry_delay is how long the scheduler waits between a try that raised and the next one, a timedelta that defaults to 300 seconds. With retries=2 a task can run 3 times in total; the count names the extra tries, never the first one.

When load raises, the worker reports the failure along with the try number, 1, and the report is checked against the task's retries, 2. Tries remain, so the cell is recorded as up_for_retry instead of failed, along with the time of the raise. For the next 5 minutes the cell sits in up_for_retry: no worker holds it, nothing of the task is executing, and the grid shows the cell in the waiting colour. The scheduler's part is the wait: once retry_delay has elapsed it moves the cell back to scheduled, then queued, and a worker takes it to running as try number 2. In the night below, try 2 returns, the cell ends success, and the run ends success about 8 minutes later than it would have without the outage. Had try 2 raised as well, the same loop would run once more, and try 3 would be the last one retries=2 allows: a raise there is recorded as failed, which is final.

Set the pair once on the DAG with default_args={"retries": 1, "retry_delay": timedelta(minutes=2)} and every task inherits it; a task that passes its own retries overrides the inherited value for itself only. The order Airflow reads is the task's own argument, then default_args, then its built-in default. In the DAG above, extract and transform carry the built-in 0, so a raise in either of them is final on the first try.

The worst case a retry setting creates is the time until the cell is finally failed: every try's running time plus every wait. load runs about 3 minutes before it raises, so retries=2 with a 5 minute delay means 3 tries of 3 minutes and 2 waits of 5 minutes, 19 minutes from the first start to a red cell. Pick the delay from the outage you are covering: a warehouse that drops connections for a minute at a time is served by a short delay, and a service that restarts over 10 minutes needs a delay longer than that or every try lands inside the same outage.

Every task so far waited for the one above it. The rule behind that waiting is the default rule for every task: a cell moves to scheduled only when every cell it depends on has ended success. The rule has a second half. When an upstream cell ends failed, the condition can never be met, so the scheduler does not leave the dependent cell waiting for the rest of the night: it writes upstream_failed straight into it, without ever sending it to a worker. That cell is now final, so the cells that depend on it get the same treatment, and the state walks down the chain until it reaches a task with nothing below it.

The daily_report DAG below has 2 tasks feeding a third. pull_sales and pull_returns both run when the run appears, because neither depends on the other. build_report depends on both, and send_report depends on build_report. When pull_returns raises with no retries, the scheduler writes upstream_failed on build_report the moment it sees the red cell, then on send_report. pull_sales keeps running and ends success, because its own upstream condition has nothing to do with the failure.

Spreading follows the arrows and nothing else. Add an archive_sales task that depends on pull_sales alone, and on the night pull_returns raises it still runs and ends success, because none of the cells above it failed. The run still ends failed, since 1 of its cells is failed, but the work that never needed the returns file is done. The arrows you draw decide how much of a bad night survives: a task wired below every other task loses whenever any of them fails.

A task can also end without failing and without doing anything. pull_returns reads a returns file that the finance system produces only on days with returns. On a day with none, the task raises AirflowSkipException, and the worker reports skipped. Under the same default rule, skipped is a final state other than success, so build_report becomes skipped as well, and then send_report. The shape of the spread is identical to a failure. The meaning is the opposite: nothing went wrong, and there was nothing to do.

The 2 grids look alike and the runs end differently. On the left, 1 cell is failed, so the run ends failed, the DAG's run list shows a red run, and anyone watching for failed runs hears about it. On the right, no cell is failed: the run ends success with 3 grey cells inside it. A report that was never built and never sent sits under a green run. That is correct by the rule, and it is why a skipped chain deserves a check of its own, because the run state will never raise it.

Reading a run with a wall of orange or grey cells, the cells that spread tell you nothing about the cause, because no worker touched them. Walk up the dependencies from any upstream_failed cell until you reach a failed one; that cell had a try, and its log holds the error. Retries belong to that cell as well: a cell in upstream_failed never had a try, so retries on it has nothing to count.

The default rule can be changed per task, so a task can run after a failure above it or after a skip. A later lesson covers those rules; everything in this lesson assumes the default.

Clearing a cell erases its state. The run goes back to queued, and the scheduler sees a none cell in a run whose other cells are final, so it does what it does with any none cell: checks the upstream condition, writes scheduled, and hands the task to the worker pool. The old try is never replayed. The task's code runs again from the start, as a new try with the next try number. Clearing works on any final cell, failed, upstream_failed, skipped and success alike, which is why it is the one tool you reach for both after a fix and when a green cell wrote the wrong thing.

Downstream is the box that decides whether the run recovers. Take nightly_orders on a night when transform raised with no retries: transform is failed and load is upstream_failed. Clear transform alone and it reruns and ends success, and then nothing else happens, because load is still a final cell and the scheduler does not revisit final cells. The run ends failed a second time. Clear transform with Downstream ticked and load goes back to none in the same action, waits for transform to end success, and runs. Switch the setting under the night below to watch the 2 outcomes.

The same action from a terminal: airflow tasks clear nightly_orders -t transform -s 2026-03-09 -e 2026-03-09 --downstream --yes picks the task by a regular expression on its id, the runs by a range of logical dates, and --downstream ticks the box. The --only-failed flag is the Only Failed box, and leaving out --yes prints the list of cells about to be cleared and asks first, which is the form to use on a range wider than 1 day.

Past and Future reach along a task's row, across runs, where Upstream and Downstream reach down one run's column. Clear transform in the 2026-03-09 run with Future ticked, and its cells in the 2026-03-10 and 2026-03-11 runs are cleared too, if those runs exist, and each one reruns against its own window. That is the box for a code bug that wrote bad output on several nights in a row. On a one-night failure such as a dropped connection, it reruns later cells that were already right, along with everything Downstream adds below each of them.

Clearing a success cell is the same action with a different consequence. The scheduler reruns the task, and with Downstream ticked, everything after it. A load that appends the window's rows to a table appends them a second time, and a transform that writes a file by date overwrites it with the same content. Before clearing a green cell, read what the task writes and ask whether running it twice leaves the data the same as running it once. A later lesson is about writing tasks so the answer is always yes.

Try numbers keep counting across a clear. A load that raised on tries 1, 2 and 3, was cleared, and raised again shows try 4 in its cell, and all 4 logs stay where they were written. The next section is about those logs.

Every try writes its own log file, and nothing overwrites it. The worker opens the file as the cell turns running, writes a header naming the task instance and the try number, streams every line the task's code logs or prints, and closes with the state it reported. A cell that reached success on try 3 therefore has 3 files, and the 2 earlier ones hold 2 tracebacks that the green cell never mentions.

The default path under the log folder spells out the 4 names from the first section: dag_id=nightly_orders/run_id=scheduled__2026-03-09T00:00:00+00:00/task_id=load/attempt=2.log. The attempt segment is the try number, so the files of one cell sit side by side in one folder, and a listing of that folder is a history of the cell.

The task instance panel's Logs tab opens on the latest attempt and offers the others in a selector. The latest one tells you how the story ended: the final traceback for a failed cell, or the normal output for a cell that recovered. The first one tells you how it started, with the original error and its timestamp, which matters when a later try failed for a different reason, such as a lock left behind by try 1. Read the last, then the first, then compare.

Take a load cell that is red after 3 tries, with retries=2 and a 5 minute delay. attempt=3.log ends on a TimeoutError raised while waiting for a lock on the orders table, and read alone it points at a slow warehouse. attempt=1.log tells a different story: try 1 wrote half its rows, then raised ConnectionResetError at 00:08 when the connection dropped. The warehouse had not yet noticed the dead connection, so the transaction try 1 opened still held the table lock when tries 2 and 3 started, and both timed out waiting on it. The fix belongs in how load handles a dropped connection; a longer retry_delay would only have hidden the cause.

A task can tell its own log which try it is in, which turns a stack of similar files into a readable sequence. get_current_context() returns the running cell, and its try_number is the same number the grid shows.