Task Lifecycle: Advanced

retries and retry_delay give every failed try the same wait, and a fixed wait fits one kind of failure: a service that blinks for 20 seconds. A rate limit that lifts after 10 minutes, or a warehouse queue that drains over half an hour, outlasts a fixed policy. With 3 retries 1 minute apart, every retry lands inside the outage, so the task fails after roughly 3 minutes of waiting with 4 identical tracebacks. This section builds a policy that spaces its tries out, and gives up at once on the errors no wait can fix.

retry_exponential_backoff=True makes each wait longer than the one before it. Airflow computes the wait at the moment a try fails. The base is retry_delay multiplied by 2 raised to one less than the number of the try that failed, so the first wait is 1 base, the second 2 bases, the third 4. On top of the base it adds a jitter of between 0 and the base, taken from a hash of the DAG id and task id combined with the logical date and the try number. The hash is fixed for one task instance and differs across instances, so 200 mapped tasks that hit one rate limit together come back spread over a window rather than as a single wave that trips the limit again. max_retry_delay caps the result, jitter included.

Add the last column and the fifth try starts between 65 and 90 minutes after the first failure, where a fixed 5 minute policy would have spent all 4 retries in 20 minutes. The wait is measured from the moment the try ended, so a try that runs for 20 minutes and then fails comes back 25 to 30 minutes after it began, which matters when a deadline downstream is close. In the grid the cell sits in up_for_retry for the whole wait: the scheduler stores the try's end time plus the wait as the earliest start and ignores the task until the clock reaches it. on_retry_callback fires as the cell enters that state, once per failed try, which is where a counter of how often each source needs a second try belongs.

A retry assumes the next try can end differently. A revoked API key, a file whose header names a column the warehouse lacks, a query with a typo in it: 5 tries of those cost up to 90 minutes, produce one traceback 5 times, and the person on call hears about it up to 90 minutes late. Airflow lets the task say that a wait would be wasted. Raising AirflowFailException moves the task instance straight to failed, however many retries remain. Any other exception moves it to up_for_retry while retries remain and to failed when they run out. The task body classifies the error it caught and raises the kind that fits.

Status codes 400 and 404 say the request itself was wrong, and so do 401 and 403; the next try would send the same request. A 429 is also a 4xx, but it says the request was fine and the caller should come back later, so the code leaves it off the list. It falls through to raise_for_status() with the 5xx answers, which say the server was wrong, and raises an ordinary exception; a timeout raises one from requests.get itself, and in each case the backoff policy takes over. For a source that has nothing to give today, AirflowSkipException ends the try as skipped, and the try is neither retried nor failed. A try that overruns execution_timeout is killed and counted as a failed try, so a task with 4 retries and a 10 minute timeout can hold a worker for 50 minutes of running time before it fails for good.

A DAG that works in a scratch table runs 4 tasks in one chain. The first 2 create the table and load it; the last 2 roll it up and drop it. The chain has a hole in it. When load_orders fails, drop_tmp sits downstream of a failure and goes upstream_failed, so the table stays until a person notices. On a scratch cluster the same hole bills by the hour all night. The repair the trigger rules lesson suggests, trigger_rule="all_done" on the drop, opens a second hole: the run's state follows its leaf tasks, drop_tmp is the only leaf, and once it succeeds the run turns green with a failed load inside it, so the failure callback never fires.

@setup and @teardown replace @task on the 2 tasks that frame the work, and the with block wires everything created inside it after the setup and before the teardown, so the 4 tasks form the same chain as the plain version. The marking changes 3 things about drop_tmp. Its trigger rule becomes all_done_setup_success: it runs once every upstream task has finished, in any state, provided the setup it belongs to succeeded. Its own state stops counting toward the run's state, so a green teardown under a failed load leaves the run failed. And clearing any task between the pair clears the pair with it, so a rerun of aggregate recreates the table before it and drops it after.

Play the night below. A manual trigger at 02:02 creates the run; create_tmp runs for 2 minutes and succeeds; load_orders runs for 4 minutes and fails; aggregate goes upstream_failed without starting; and drop_tmp, with its setup green and every upstream cell settled, is scheduled, runs for 1 minute and succeeds. The run closes as failed with the teardown green inside it. Switch the variant to a night where the load succeeds and the only difference is the 2 cells in the middle.

The last 2 rows show the default for a teardown that fails on its own: the work succeeded, the run reports success, and the undropped table shows only as the teardown's red cell. For a cluster that bills until it is deleted, that default hides the cost. @teardown(on_failure_fail_dagrun=True) makes the teardown's failure fail the run, which routes it to the same on_failure_callback as any other failure. Choose per teardown: a temp table nobody pays for can stay a red cell, a cluster cannot.

Operators from providers take the same marking through 2 methods. create.as_setup() returns the task marked as a setup, and delete.as_teardown(setups=create) returns the teardown tied to it. setups takes a list when one teardown closes several setups, and the teardown then waits for all of its upstream tasks and runs when its setups succeeded. The work tasks between the pair need no marking at all; the chain through the teardown places them.

Clearing follows the pair as a unit. Clear run_job after fixing its bug and the grid clears create_cluster and delete_cluster with it, because the rerun needs a cluster that exists and must delete it afterwards. Clear delete_cluster alone and only the delete reruns, which is the move when the job finished and the delete was the task that failed. A clear of the setup alone also clears the work and the teardown, since work that ran on the old cluster cannot be kept once a new one is made.

When approve_prices starts, it records the question and leaves the worker: the cell turns to deferred, a state that means the task is waiting on an event outside any worker, with its wait held by the triggerer, a long-lived Airflow process beside the scheduler. A later lesson covers that state in depth; for now, read it as a wait that costs no worker slot, so 30 runs waiting on 30 analysts occupy nothing. The question appears on the Required Actions page of the UI, showing the subject and body above its 2 buttons, next to the run it belongs to. Any user with access to the DAG can answer, unless assigned_users names who may. The task instance stores the answer along with who gave it and when.

Reject ends the task in success, because the task did its job of asking, and skips the tasks after it. By default ignore_downstream_trigger_rules is False: it skips the tasks directly downstream, here publish_prices. Tasks further down follow their own trigger rules, so a notify_team after publish_prices with trigger_rule="all_done" still runs on a Reject. Setting it to True skips every task downstream whatever its trigger rule, as a short-circuit does by default. fail_on_reject=True makes Reject fail the task instead, for a run that should turn red when the answer is no. Approve returns and the chain continues. The option the person chose is stored with the task, so a task further down can read which answer was given, which belongs to the lessons on passing values between tasks.

execution_timeout bounds the wait, the same argument that bounds any task. With defaults set, the timeout picks that answer and the task ends as if a person had chosen it, which is the shape for a decision with a safe side: an unreviewed price list stays unpublished. Without defaults, the timeout fails the task, and the retry policy from the first section applies to it like any failure, so retries=1 asks the question a second time 4 hours later. Clearing the task from the grid asks again at once, with a fresh 4 hours.

HITLBranchOperator offers a list of task ids as its options and runs the one the person picks, the branching pattern from the intermediate lesson with a person in place of the function. HITLOperator takes any list of options, and multiple=True lets the person pick several. HITLEntryOperator collects typed values through form fields. subject and body work the same way in all 4, and so do defaults, execution_timeout, notifiers and assigned_users.

A question on a page nobody opens waits for its timeout. notifiers takes a list of notifier objects, the kind a failure callback can be, and runs them when the task starts waiting, passing them the subject and body along with a link to the response page. A chat notifier from a provider puts the question in the analysts' channel at 05:10 with a button that opens the page, and the 4 hour clock runs from then. The timeout should match how the team works: 4 hours covers a morning, and a question posted at 02:00 with a 4 hour timeout answers itself before anyone is awake.

A task's log usually ends with a verdict. A success ends with the task's own last line and a return code of 0; a failure ends with a traceback, the exception's name, and a line from the task runner saying the task failed. Some logs end mid-sentence, with loaded 1,200,000 of 3,400,000 rows as the last line, neither a traceback nor a return code after it, and a state that stays unchanged for 5 minutes. A log that stops without a verdict means the process that would have written the verdict is gone, and Airflow has 3 ways of noticing, each leaving a different trace. The intermediate lesson's execution_timeout belongs with the failures that leave a log: the runner raises AirflowTaskTimeout inside the task, so the traceback is there and the state changes at once.

Every running task has 2 processes on its worker: the task process and a supervisor that started it. The supervisor sends a heartbeat to the API server on a short interval, [workers] min_heartbeat_interval, 5 seconds by default, and each heartbeat updates the task instance's last heartbeat time. The scheduler looks for running task instances whose last heartbeat is older than [scheduler] task_instance_heartbeat_timeout, 300 seconds by default. Each one it finds is treated as a failed try: a line naming the instance a zombie goes into its log, and the instance passes through the same failure handling a raised exception gets, so the cell goes to up_for_retry if retries are left and to failed if not. The matching callback, on_retry_callback or on_failure_callback, fires.

The 5 minutes are the price of telling a dead worker from a slow network, and the retry's wait starts after them. The gap applies only when the supervisor dies with the task: a host that loses power, an evicted pod, a container killed whole at its memory limit. A retry treats the cause as transient, which fits an evicted pod and does not fit a task that exceeds its memory limit, because the next try is killed at the same row count. With retries=3 and the default 5 minute retry_delay, one death becomes 4, each noticed 5 minutes after it happened: 20 minutes of detection plus 15 minutes of waits, 35 minutes on top of the running time before anyone is paged. The pattern to read for is a kill at the same point in the log on every try, which names the task's memory as the problem and the retry count as noise. A kill at a different point each time, on different workers, points at the hosts instead.

When the kernel's out-of-memory killer takes only the task process, the supervisor survives it, sees the child end on a signal rather than a return code, and reports the try as failed at once with a line giving the negative code, -9 for a kill. The log still has no traceback, because no Python ran after the signal. The worker's own logs, or the pod's events on a container platform, carry the reason: an OOMKilled event with the limit it hit. A termination signal instead of a kill, which is what a worker shutdown sends, lets Python run: the task raises on the signal, the log ends with a traceback naming SIGTERM, and the try fails through the ordinary path with its retries intact.

Marking a running task failed in the UI, or clearing it, changes the state in the database first. The supervisor learns of the change on its next heartbeat, because the API server answers a heartbeat for a task whose state is no longer running by telling the supervisor to stop. It terminates the task and writes that the state was set externally. The log therefore carries a verdict, written within about 5 seconds of the click, and the next try number belongs to the clear. A log that ends with that line, next to an audit log entry with a person's name, belongs to a task that someone stopped by hand.

Lowering task_instance_heartbeat_timeout below 300 seconds buys faster detection at the cost of failing healthy tasks during a slow network or a busy API server; raise it only for workers that are known to pause, and change neither without watching the zombie count for a week.

A failed run calls on_failure_callback the moment it fails. A run that is still going at 07:00, when the pricing dashboard refreshes from its output, calls nothing, because nothing has failed: pull_listings hit its rate limit at 03:10, backed off 4 times, and is on its fifth try with every other cell green or waiting. The people at the dashboard find stale prices and ask the data team at 09:00, 2 hours after the point at which a page could have changed the outcome. A deadline alert is Airflow's answer for that run: a time the run must be finished by, and a callback that runs when the clock passes it with the run still open.

The deadline is a reference plus an interval, computed once when the run is created and stored with it. DAGRUN_LOGICAL_DATE anchors to the run's logical date, and with CronTriggerTimetable that is the 02:00 the run fired at, so the deadline is 07:00 the same morning for every scheduled run. With CronDataIntervalTimetable the logical date would be the previous day's 02:00, and the same 5 hours would name a deadline already past when the run is created, so the alert fires as the run starts; the timetable lesson's distinction decides whether the alert means anything. DAGRUN_QUEUED_AT anchors to the moment the run was created, which measures how long the run took regardless of when it was due, and FIXED_DATETIME names one instant for a run that must land before a specific meeting.

The scheduler checks pending deadlines on each loop, and a deadline whose run is still open when the clock passes it fires once. AsyncCallback names an async def in a module the triggerer can import, and the triggerer runs it with the kwargs you set alongside details of the run that missed, so the page can link to the grid. Running it in the triggerer matters because a late run is often late for lack of workers; a callback that needed a worker slot would queue behind the tasks it is reporting on. The alert fires whether the run then succeeds at 07:25 or fails at 09:00, since what it measures is lateness, and a run that fails before 07:00 reaches a terminal state and never fires it.

Set the interval from the consumer's clock, then subtract the retry policy's worst case. A deadline at 07:00 with pull_listings able to wait 90 minutes across its retries means a first failure after 05:30 cannot finish in time, which is a fact to carry back to the backoff section: cap the waits so the last retry still fits. A DAG whose runs are late on most nights has a capacity problem, and the alert will report it every morning until the platform has room for that DAG; the alert only reports the lateness, and the fix lies in the platform's capacity.