**kwargs for Keywords
Concepts covered: Flexible Arguments
A parameter written with 2 stars, **options, gathers every keyword argument that no named parameter matched and stores them in a dictionary. Each key is the argument's name as a string, each value is what the caller passed, and the order is the order of the call. With it, a function accepts settings whose names it doesn't know in advance, such as the tags on a job or the options a writer passes on to storage.
def build_job(name, **settings):
print("job:", name)
print("settings:", settings)
for key, value in settings.items():
print(" ", key, "=", value)
build_job("daily_orders", retries=2, owner="ana", priority="high")
build_job("hourly_clicks")Output:
job: daily_orders
settings: {'retries': 2, 'owner': 'ana', 'priority': 'high'}
retries = 2
owner = ana
priority = high
job: hourly_clicks
settings: {}name is matched first, so it never appears in settings. With no extra keywords, settings is the empty dict {} and the loop runs 0 times. Because settings is an ordinary dictionary, the .items() loop you already use for dicts works on it unchanged.
Named parameters claim their keywords first
When the function also declares a parameter by name, a keyword with that name goes to the parameter, and the dictionary never sees it. A wrapper can pass its own dictionary along with ** in a call, which spreads the dictionary back into keyword arguments:
def write_table(table, mode="append", **options):
print("write", table, mode)
for key, value in options.items():
print(" option", key, value)
def run_step(table, **options):
print("step start:", table)
write_table(table, **options)
run_step("orders", mode="overwrite", compression="zstd", partitions=8)step start: orders write orders overwrite option compression zstd option partitions 8
Watch the 2 options dictionaries. In run_step it holds 3 keys, because run_step names none of them. The call write_table(table, **options) spreads them back into mode="overwrite", compression="zstd", partitions=8. mode matches a named parameter, so the options inside write_table holds only the other 2. The wrapper forwards settings it never inspects, so adding a writer option needs no change to run_step.
Spreading a dict that does not fit
Spreading a dictionary with ** checks every key against the function's parameters. A function without its own ** parameter has nowhere to put a key it does not name:
What does this program print?
def connect(host, port=5432):
print(host, port)
settings = {"host": "db-1", "port": 6543, "timeout": 30}
connect(**settings)- It prints db-1 6543, and timeout is ignored because connect has no use for it
- It prints db-1 5432, because the default wins when the dict has extra keys
- It raises TypeError: connect() missing 1 required positional argument: 'host'
- It raises TypeError: connect() got an unexpected keyword argument 'timeout'
The opposite clash fails too. connect("db-2", host="db-1") supplies host twice, once by position and once by name, and raises TypeError: connect() got multiple values for argument 'host'. Python refuses to choose between 2 values for one parameter.
The full signature order
All 4 kinds of parameter can appear in one def. This lesson writes them in this order, which is the most common one:
| Position | Parameter | What it receives |
|---|---|---|
| 1 | `table` | A required argument, by position or by name |
| 2 | `mode="append"` | An optional argument with a fallback |
| 3 | `*paths` | Every leftover positional argument, as a tuple |
| 4 | `**options` | Every leftover keyword argument, as a dict |
def ingest(table, mode="append", *paths, **options):
print(table, mode, paths, options)
ingest("events", "overwrite", "p1", "p2", compression="zstd")
ingest("clicks")Output:
events overwrite ('p1', 'p2') {'compression': 'zstd'}
clicks append () {}Positional arguments fill table, then mode, then spill into paths. Keyword arguments match parameter names first, then spill into options. **options must always be the last parameter, so a def that places it before *paths is a SyntaxError. The other positions allow more than the table shows. A parameter written after the star, as in def ingest(table, *paths, mode="append", **options):, is valid: *paths takes every leftover positional argument, so mode can only be set by name, as in mode="overwrite". That's the third shape promised for the copy_files bug, and a later lesson covers it.
- Do
- Give each option the function reads its own named parameter
- Use **options when the function forwards settings it does not read
- Keep the order: required, defaulted, *args, **kwargs
- Avoid
- Don't spread a dict into a function unless every key names a parameter
- Don't use **options to avoid writing a clear signature
- Don't forward one **options to 2 functions that accept different names
About this lesson
This section is part of the Functions: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.