Branching and the Skipped Join

Concepts covered: Trigger rules and branching

A branch task returns the task id of the path to take, or a list of ids when more than 1 path should run. Airflow then sets each task directly downstream of the branch that was not returned to skipped, and the skip spreads: a task behind a skipped task under all_success is skipped as well, all the way down the DAG. The decorator form is @task.branch; BranchPythonOperator does the same from a python_callable. The function can read anything in the run's context, and most branches check logical_date: the 1st of the month rebuilds, every other day appends.

from datetime import datetime

from airflow.sdk import dag, task


@dag(schedule="@daily", start_date=datetime(2026, 11, 1), catchup=False)
def orders_load():
    @task.branch
    def pick_load_path(**context):
        if context["logical_date"].day == 1:
            return "load_full"
        return "load_delta"

    @task
    def load_full():
        print("orders table rebuilt from scratch")

    @task
    def load_delta():
        print("one day of orders appended")

    @task(trigger_rule="none_failed_min_one_success")
    def publish():
        print("orders table published")

    pick_load_path() >> [load_full(), load_delta()] >> publish()


orders_load()

On the run for 1 November the branch returns load_full, so load_delta is marked skipped and load_full runs. On the run for 15 November it returns load_delta and load_full is the one skipped. Either way publish sees 1 succeeded upstream and 1 skipped upstream. With the default rule in place of the one above, that's not enough: a skipped upstream leaves all_success unmet, and the scheduler sets publish to skipped. The orders table is loaded and never published, and the run badge is green, because its only leaf ended in skipped and a skipped leaf counts toward success.

Choosing the join's rule

none_failed_min_one_success is the rule a join wants: it runs when no upstream failed and at least 1 succeeded, so the skipped path is ignored and a failed path still blocks the publish. none_failed is close, with 1 difference that matters: it also runs when every upstream was skipped, which is what happens when a branch returns an empty list to say that nothing should load this run. all_done would run publish after load_full failed, announcing a table that was never rebuilt. one_success starts the join the moment 1 path succeeds, which is fine with 2 exclusive paths and wrong once the branch can return 2 ids, because the join would begin while the second path is still loading. When the chosen path fails, every one of these rules except all_done leaves publish in upstream_failed and the run in failed, which is the outcome you want.

from datetime import datetime
from airflow.sdk import dag, task
@task.branch
def pick_load_path(**ctx):
    if ctx["logical_date"].day == 1:
        return "load_full"
    return "load_delta"
@task
def load_full(): ...
@task
def load_delta(): ...
@task(trigger_rule="all_success")
def publish(): ...
@dag(schedule="@daily",
     start_date=datetime(2026, 11, 1))
def orders_load():
    loads = [load_full(), load_delta()]
    pick_load_path() >> loads
    loads >> publish()
orders_load()

The branch skips only its direct downstream

The branch itself touches only its direct downstream. A task wired directly behind the branch whose id the branch never returns is skipped on every run. Say someone hangs notify_which_path off pick_load_path to report the choice: the branch returns load_full or load_delta, never notify_which_path, so that cell is grey every night and nothing is reported. Put that task behind the join instead, where the join's rule decides, or return its id from the branch along with the chosen path. The branch's own cell is success on every run, including a run where it chose nothing; the choice it made is in its log, and the grid shows it only through which cells turned grey.

About this lesson

This section is part of the Task Lifecycle: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.