Passing Values with XCom

Concepts covered: Passing data between tasks

XCom, short for cross-communication, is the channel one task uses to hand a small value to another task in the same run. The value is a row in the xcom table of Airflow's metadata database. Airflow finds that row by the run id and task id it belongs to, together with the DAG id and a key. When sum_payouts finishes, Airflow writes that row; when notify starts, on whatever worker it lands, it reads the row back. The 2 processes share only that row, which is why the value has to be small enough to sit in a database and plain enough to serialize.

from datetime import datetime, timedelta
from airflow.sdk import dag, task

@dag(schedule=timedelta(days=1), start_date=datetime(2026, 3, 1))
def host_payouts():

    @task
    def sum_payouts():
        job = warehouse.submit("INSERT INTO host_payouts SELECT ...")
        job.wait()
        return job.rows_written

    @task
    def notify(paid_hosts: int):
        print(f"{paid_hosts} hosts paid")

    notify(sum_payouts())

host_payouts()

In the last line of that DAG function, passing the result of sum_payouts() into notify(...) draws the edge from sum_payouts to notify, the same dependency >> would draw. The same line also moves the value at run time: the integer sum_payouts returns is pushed to the xcom table under the key return_value, and when notify begins its try, Airflow pulls that row from the same run and passes it in as paid_hosts. The function bodies never mention XCom, because the return statement and the argument stand in for the push and the pull. A task that returns None pushes nothing.

Pushing and pulling by hand

@task
def sum_payouts(ti=None):
    job = warehouse.submit("INSERT INTO host_payouts SELECT ...")
    job.wait()
    ti.xcom_push(key="job_id", value=job.id)
    return job.rows_written

@task
def notify(ti=None):
    hosts = ti.xcom_pull(task_ids="sum_payouts")
    job_id = ti.xcom_pull(task_ids="sum_payouts", key="job_id")
    print(f"{hosts} hosts paid by job {job_id}")

sum_payouts() >> notify()

ti is the task instance, which Airflow hands to any @task function that declares a ti parameter. xcom_push writes a row under the key you choose, so one task can publish 2 values: the count under return_value and the job id under job_id, which adds a second row per run to the table above. xcom_pull reads by task id and key, and the key defaults to return_value, so the first pull in notify fetches the count. A pull for a key nobody pushed returns None rather than raising, so a missing push surfaces as a None further down. Because nothing is passed as an argument now, the dependency has to be drawn with >>; without it, notify could start first and pull a row that does not exist yet.

The 4 rules of an XCom value

• JSON only. A dict, list, string, number or boolean serializes; a dataframe or a database cursor fails at push time with a serialization error, and the task fails with it.
• Small. A count, a path, a job id, a list of 20 file names. Every run of every DAG adds its rows to the same table, in the same database the scheduler uses.
• Scoped to the run. notify in the 2026-03-02 run pulls the 2026-03-02 row and cannot see the 2026-03-01 one, so the value a task reads came from this run's upstream.
• Wiped per try. When a task instance starts a try, its own XComs from earlier tries are deleted first, so a retry never reads a half-result left by the try that failed.

Predicting what a rerun reads

Say the 2026-03-01 run succeeded with sum_payouts pushing 41872. A week later someone clears notify alone to resend the message. notify reruns and pulls 41872, the same row, because the row belongs to sum_payouts and that task was not touched. Now say they clear sum_payouts instead. Clearing it deletes its XCom rows along with its state, and notify is cleared with it as a downstream task. When sum_payouts reruns it pushes a fresh count, 41880 if 8 late reservations landed in the meantime, which notify then pulls. In both cases the XCom row exists exactly as long as the task instance that pushed it keeps its state.

About this lesson

This section is part of the Tasks and Data: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.