*args for Variadics

Concepts covered: Flexible Arguments

Some functions take a number of inputs that varies per call: the file sizes in one partition, the columns to check for nulls, the tables to refresh. A parameter written with a star, *sizes, gathers every positional argument that no earlier parameter took and stores them as a tuple. The caller passes the values separately and never builds a container.

def total_bytes(*sizes):
    print("received:", sizes)
    total = 0
    for size in sizes:
        total += size
    return total

print(total_bytes(120, 340, 40))
print(total_bytes(75))
print(total_bytes())

Output:

received: (120, 340, 40)
500
received: (75,)
75
received: ()
0

The tuple can have any length. One argument gives (75,), where the trailing comma marks a 1-value tuple. No arguments gives the empty tuple (), which the loop walks 0 times, so the total stays 0. The name after the star is a convention: *args is the common generic name, and a descriptive one like *sizes documents what belongs there.

Fixed parameters come before the star

Named parameters still work ahead of a starred one. Python fills them first, left to right, and the star takes whatever is left:

def log(level, *messages):
    for message in messages:
        print(level, message)

log("WARN", "late partition", "schema drift")
log("INFO")

Output:

WARN late partition
WARN schema drift

log("INFO") prints nothing: level takes the only argument and messages is empty. A defaulted parameter placed before the star is filled the same way, which causes a common surprise:

What does this program print?

def copy_files(dest, retries=3, *files):
    print("dest:", dest)
    print("retries:", retries)
    print("files:", files)

copy_files("archive", "a.csv", "b.csv")

The bug raises no error, which makes it expensive: the job copies only b.csv, and later code treats 'a.csv' as a retry count. With what you know now, there are 2 safer shapes. Leave retries out of a function that takes a variable number of files, or take the files as one list parameter so retries can stay optional. A later lesson adds a third shape that fixes this inside the signature.

Take *files
  • Callers usually write the values out: copy_files("archive", "a.csv", "b.csv")
  • The count changes from call to call
  • No defaulted parameter needs to sit before the star
Take one list parameter
  • Callers usually hold the values in a list already
  • Other parameters can keep defaults without being captured
  • The call shows where the group of values starts and ends

Spreading a list into a call

The star also works in the other direction. In a call, * in front of a list spreads its items into separate positional arguments:

def total_bytes(*sizes):
    total = 0
    for size in sizes:
        total += size
    return total

partition_sizes = [120, 340, 40]
print(total_bytes(*partition_sizes))
print(total_bytes(*partition_sizes, 500))
print(*partition_sizes)

Output:

500
1000
120 340 40

Inside the function, sizes is (120, 340, 40) in the first call and (120, 340, 40, 500) in the second. Calling total_bytes(partition_sizes) without the star makes sizes a tuple holding one list, and the loop's total += size fails with TypeError: unsupported operand type(s) for +=: 'int' and 'list'.

Star written inDirectionExample
The `def` lineMany arguments gathered into one tuple`def total_bytes(*sizes):`
A callOne list spread into many arguments`total_bytes(*partition_sizes)`

About this lesson

This section is part of the Functions: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.