Short-Circuits and LatestOnly

Concepts covered: Trigger rules and branching

A short-circuit task runs a function and reads its return value as a verdict. A truthy value lets the run continue. A falsy value skips every task downstream of the short-circuit, direct or not, whatever trigger rule each carries, and the short-circuit itself ends in success. The run finishes green with most of its column grey, which is the right shape for an hour in which there was nothing to do. The decorator form is @task.short_circuit; ShortCircuitOperator takes a python_callable and does the same.

from datetime import datetime

from airflow.sdk import dag, task
from clicks_landing import landed_click_count


@dag(schedule="@hourly", start_date=datetime(2026, 11, 1), catchup=False)
def hourly_clicks():
    @task.short_circuit
    def clicks_arrived(**context):
        landed = landed_click_count(context["data_interval_start"])
        return landed > 0

    @task
    def aggregate_clicks():
        print("hour aggregated")

    @task
    def refresh_dashboard():
        print("dashboard refreshed")

    clicks_arrived() >> aggregate_clicks() >> refresh_dashboard()


hourly_clicks()

On an hour with 0 landed clicks, clicks_arrived returns False and ends in success, and aggregate_clicks and refresh_dashboard are set to skipped by the short-circuit itself, before the scheduler evaluates any rule. That reach past the direct downstream is the parameter ignore_downstream_trigger_rule, which defaults to True. Set it to False and the short-circuit skips only aggregate_clicks; refresh_dashboard is then left to its own rule, so under all_success it is skipped as well, and under none_failed it runs on the empty hour. On an hour with clicks, the function returns True, nothing is skipped, and the verdict cost 1 short task.

An early return inside aggregate_clicks would give the same result for less code, and a worse grid. With the separate check, the empty hour is visible as grey cells, the 2 tasks behind it never take a worker, and a task wired behind them with ignore_downstream_trigger_rule=False and none_failed can still post the message that nothing landed.

LatestOnly: rebuild history, refresh once

LatestOnlyOperator answers 1 question when its task runs: is this run the newest one the schedule has produced? It takes the run's window, asks the timetable for the window after it, and checks whether the current clock time falls inside that next window. If it does, this run is the latest and the operator lets every downstream task proceed. If the clock has moved past that next window, a newer run exists or is due, and the operator skips its direct downstream; from there the skip cascades through trigger rules in the usual way, so a task behind the gate under none_failed would still run.

from datetime import datetime

from airflow.providers.standard.operators.latest_only import LatestOnlyOperator
from airflow.sdk import dag, task


@dag(schedule="@daily", start_date=datetime(2026, 10, 1), catchup=True)
def daily_prices():
    @task
    def build_price_history():
        print("one day of prices written")

    @task
    def refresh_price_cache():
        print("live price cache refreshed")

    latest_only = LatestOnlyOperator(task_id="latest_only")

    build_price_history() >> latest_only >> refresh_price_cache()


daily_prices()

Say the DAG was paused for 3 days and unpaused at 00:10 on 2 November. Catch-up creates the 3 missing runs, and each reaches latest_only by 00:15. The operator works the same arithmetic in all 3, against the same clock:

build_price_history ran in all 3, so 3 days of history were written; refresh_price_cache ran once, from the run whose next window holds the clock. The 31 October run lost by 15 minutes: its next window closed at 2 November 00:00, and the check happened after that. A run you trigger by hand is exempt: the operator reads the run's type, sees manual, and lets everything proceed, because a person asked for that run. A run created by a backfill carries the backfill run type instead, so the check applies and old dates skip the refresh, which is the reason to place the operator ahead of anything that only makes sense once.

Short-circuit

• Verdict comes from your function's return value
• Skips everything downstream by default
• The skip ignores trigger rules unless you turn that off
• Fires on any run, including a manual one

LatestOnlyOperator

• Verdict comes from the clock and the next window
• Skips its direct downstream only
• The skip cascades through trigger rules
• Lets a manual run through untouched

Do

• Put the short-circuit first, so an empty hour costs 1 small task and 0 workers downstream
• Place latest_only directly ahead of the task that must happen once, after the history writers
• Read the verdict in the short-circuit's log when a grey column surprises you

Avoid

• Put latest_only at the top of a DAG you plan to backfill, or the backfill writes nothing
• Rely on all_done behind a short-circuit while expecting the empty hour to skip it
• Return None from a short-circuit by accident; None is falsy and skips the run

About this lesson

This section is part of the Task Lifecycle: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.