Grouping Tasks

Concepts covered: Task dependencies

A DAG with 30 tasks draws a graph nobody reads. If 12 of those tasks pull from source systems, 12 clean what arrived and 6 publish, the graph has 3 shapes in it, and a TaskGroup lets you draw those 3 shapes as 3 nodes. The group is a naming and drawing device: it changes task ids and how the UI folds the graph, and leaves what each task runs untouched.

from airflow.sdk import DAG, TaskGroup, task

with DAG(dag_id="orders_daily", schedule=None) as dag:
    with TaskGroup("extract") as extract:
        @task
        def pull_orders():
            print("orders")

        @task
        def pull_customers():
            print("customers")

        @task
        def pull_payments():
            print("payments")

        pull_orders()
        pull_customers()
        pull_payments()

    with TaskGroup("clean") as clean:
        @task
        def dedupe():
            print("dedupe")

        @task
        def cast_types():
            print("cast")

        dedupe() >> cast_types()

    extract >> clean

Each task inside a group got the group's name prefixed to its id, so the task you wrote as pull_orders is stored and shown as extract.pull_orders, and that full id is what every command takes, so testing it is airflow tasks test orders_daily extract.pull_orders 2026-11-03. extract >> clean also wired the groups: every task in extract with nothing downstream inside its group now points at every task in clean with nothing upstream inside its group. Here that is 3 tasks pointing at 1, since cast_types already has dedupe upstream, so the single arrow created 3 edges.

The decorator form

from airflow.sdk import DAG, task, task_group

with DAG(dag_id="orders_daily", schedule=None) as dag:
    @task_group
    def extract():
        @task
        def pull_orders():
            print("orders")

        @task
        def pull_customers():
            print("customers")

        pull_orders()
        pull_customers()

    @task_group(group_id="clean", tooltip="dedupe then cast")
    def clean_stage():
        @task
        def dedupe():
            print("dedupe")

        dedupe()

    extract() >> clean_stage()

@task_group wraps a function the way @dag wraps a DAG: the group exists when the function is called, and the call returns an object you can wire with >>. The decorator takes the same arguments as TaskGroup. tooltip shows when the pointer rests on the collapsed node. prefix_group_id=False keeps the inner ids bare, at the cost of making you keep them unique across the whole DAG yourself.

Arrows need not stop at group borders. clean >> notify, with notify a plain task, hangs that task off every leaf of the group, and an inner task can be wired on its own: keep the return value of pull_orders() in a variable and write orders >> some_task to join 2 tasks in different groups without touching the rest of either group.

What the UI does with a group

In the graph, a group draws as one node that opens on a click to show the tasks inside, and closes again. In the grid, a group is one row with a chevron, and the row's color summarizes its tasks: the row goes red if any task inside failed, and expanding it shows which one. Nesting works the same way at each level, so a DAG of 60 tasks can open as 4 nodes, each opening into 3, each opening into 5.

The rename nobody expects

Putting existing tasks into a group renames them. pull_orders becomes extract.pull_orders, and the metadata database has task history under pull_orders and nothing under the new id. The UI shows the old id as a removed task and the new one with an empty history, and any script that calls airflow tasks test with the old id now fails to find it. For a DAG with a long history worth keeping, prefix_group_id=False on the group gives you the folded graph with the ids unchanged. For a new DAG, let the prefix happen and keep group names short, since every task id carries them from then on.

Do

• Group by purpose: extract, clean, publish, each with a one-word id
• Wire groups with one >> and let Airflow fan the edges out
• Use the full dotted id in every command and search
• Nest when a group passes 10 tasks of its own

Avoid

• Group tasks that share nothing except position in the file
• Switch off the prefix on a new DAG to save typing
• Rename live tasks by grouping them without telling whoever reads their history
• Expect a group to change how or when its tasks run

About this lesson

This section is part of the DAG Anatomy: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.