Commands, Not Data

Concepts covered: Scheduler and executors

Every task Airflow starts is a Python process on a worker, a machine with a fixed amount of memory shared by every task running on it. The warehouse that holds the reservation rows was built to hold them: it keeps them on its own disks with indexes over them, and its query engine sums 60 million rows across many cores without loading them into any one process. A task that copies those rows over the network into its own memory discards all of that and takes on the job with a single core and a few gigabytes. A task that asks the warehouse for the sum and waits for the answer leaves the summing to the engine built for it. This lesson is built on that division: Airflow decides when and in what order, and the systems that own the data do the moving.

2 ways to write the same task

from airflow.sdk import task

@task
def carry_the_rows():
    rows = warehouse.fetch(
        "SELECT host_id, amount FROM reservations "
        "WHERE checkout_date = DATE '2026-03-01'"
    )
    totals = {}
    for row in rows:
        totals[row["host_id"]] = totals.get(row["host_id"], 0) + row["amount"]
    warehouse.write("host_payouts", totals)


@task
def command_the_warehouse():
    job = warehouse.submit(
        "INSERT INTO host_payouts "
        "SELECT host_id, SUM(amount) AS amount FROM reservations "
        "WHERE checkout_date = DATE '2026-03-01' GROUP BY host_id"
    )
    job.wait()
    return job.rows_written

Put sizes on the 2 versions. A reservation row of 8 columns serializes to about 120 bytes, so 60 million rows is 7.2GB crossing the network into one process, and Python dicts take several times that once the rows are loaded into them. The warehouse scans the same rows from its own disks in parallel and keeps the result, a table of 41,872 host totals, on its own disks too. The commanding task sends one statement out and gets one integer back. The carrying task pulls 7.2GB in, and its memory grows with the day's bookings: a quiet Tuesday fits, and the Saturday after a holiday fails the task on the day the figures matter most.

The difference shows up again when a try fails halfway. The commanding task retries by resubmitting one statement, and before it does, it can ask the warehouse for the state of the job the first try submitted. That statement has to be safe to run twice. If the first try's INSERT INTO host_payouts had already committed when the worker died, a plain resubmit adds the 41,872 host totals a second time, and the table holds 83,744 rows with every host listed twice. A later lesson in this track writes the statement so a rerun replaces the day's rows instead of appending them. The carrying task retries by starting the 7.2GB download from the beginning, inside a worker that may be sitting under the same memory ceiling that ended the first try.

Carrying rows through the worker

• Memory grows with the size of the day
• A retry downloads everything again
• One core does work built for a cluster
• The worker becomes the slowest machine in the pipeline

Commanding the system that owns them

• Memory holds a job handle and a count
• A retry resubmits one statement
• The warehouse uses its own cores and indexes
• The task waits, then reports a number

What Airflow keeps for itself

What Airflow stores is small and about the work rather than the data: which runs exist and what state each task is in, plus the task logs and the few values that tasks hand each other. That division decides what the rest of this lesson has to cover. For a task to command another system, it needs the address and credentials of that system, the settings that tune the command, the window of time the run covers, and a way to pass a small result to the next task. Those 4 needs map onto the 4 sections that follow. XCom carries the result and connections hold the credentials. The settings come from variables and params, and the window comes from templated fields.

The hard-coded date in both tasks is a flaw. The last section of this lesson replaces it with a value each run fills in for itself.

About this lesson

This section is part of the Tasks and Data: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.