*args for Variadics
Concepts covered: Flexible Arguments
Some functions take a number of inputs that varies per call: the file sizes in one partition, the columns to check for nulls, the tables to refresh. A parameter written with a star, *sizes, gathers every positional argument that no earlier parameter took and stores them as a tuple. The caller passes the values separately and never builds a container.
def total_bytes(*sizes):
print("received:", sizes)
total = 0
for size in sizes:
total += size
return total
print(total_bytes(120, 340, 40))
print(total_bytes(75))
print(total_bytes())Output:
received: (120, 340, 40) 500 received: (75,) 75 received: () 0
The tuple can have any length. One argument gives (75,), where the trailing comma marks a 1-value tuple. No arguments gives the empty tuple (), which the loop walks 0 times, so the total stays 0. The name after the star is a convention: *args is the common generic name, and a descriptive one like *sizes documents what belongs there.
Fixed parameters come before the star
Named parameters still work ahead of a starred one. Python fills them first, left to right, and the star takes whatever is left:
def log(level, *messages):
for message in messages:
print(level, message)
log("WARN", "late partition", "schema drift")
log("INFO")Output:
WARN late partition WARN schema drift
log("INFO") prints nothing: level takes the only argument and messages is empty. A defaulted parameter placed before the star is filled the same way, which causes a common surprise:
What does this program print?
def copy_files(dest, retries=3, *files):
print("dest:", dest)
print("retries:", retries)
print("files:", files)
copy_files("archive", "a.csv", "b.csv")- It prints dest: archive, retries: 3, files: ('a.csv', 'b.csv')
- It raises TypeError, because retries expects a number and receives a string
- It prints dest: archive, retries: a.csv, files: ('b.csv',)
- It prints dest: archive, retries: 3, files: ('b.csv',)
The bug raises no error, which makes it expensive: the job copies only b.csv, and later code treats 'a.csv' as a retry count. With what you know now, there are 2 safer shapes. Leave retries out of a function that takes a variable number of files, or take the files as one list parameter so retries can stay optional. A later lesson adds a third shape that fixes this inside the signature.
- Take *files
- Callers usually write the values out: copy_files("archive", "a.csv", "b.csv")
- The count changes from call to call
- No defaulted parameter needs to sit before the star
- Take one list parameter
- Callers usually hold the values in a list already
- Other parameters can keep defaults without being captured
- The call shows where the group of values starts and ends
Spreading a list into a call
The star also works in the other direction. In a call, * in front of a list spreads its items into separate positional arguments:
def total_bytes(*sizes):
total = 0
for size in sizes:
total += size
return total
partition_sizes = [120, 340, 40]
print(total_bytes(*partition_sizes))
print(total_bytes(*partition_sizes, 500))
print(*partition_sizes)Output:
500 1000 120 340 40
Inside the function, sizes is (120, 340, 40) in the first call and (120, 340, 40, 500) in the second. Calling total_bytes(partition_sizes) without the star makes sizes a tuple holding one list, and the loop's total += size fails with TypeError: unsupported operand type(s) for +=: 'int' and 'list'.
| Star written in | Direction | Example |
|---|---|---|
| The `def` line | Many arguments gathered into one tuple | `def total_bytes(*sizes):` |
| A call | One list spread into many arguments | `total_bytes(*partition_sizes)` |
About this lesson
This section is part of the Functions: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.