Scikit-learn's data preprocessing pipelines use functools.partial and higher-order functions to configure reusable transformation steps that work identically in development, staging, and production without any environment-specific code. Data scientists at companies like Instacart compose these partial functions into pipelines where each step receives data from the previous one, making complex ML preprocessing both testable and reproducible. The functools patterns in this lesson, including partial application and reduce(), are how professional Python developers build flexible, composable systems.
Keyword Arguments
Daily Life
Interviews
Call functions with named arguments
When calling a function, you can specify arguments by name instead of relying on position. This makes code clearer, especially for functions with many parameters or when the meaning of arguments is not obvious. Keyword arguments also let you skip optional parameters and provide only the ones you need.
Positional vs Keyword Arguments
Positional - matched to parameters by their position in the call
Keyword - matched by explicitly naming the parameter
You can mix both, but positional arguments must come first
Positional vs Keyword Args
You have been using positional arguments exclusively until now, where the order of values determines which parameter receives which value. Keyword arguments let you explicitly name the parameter you are providing a value for:
1
defformat_name(first,last,title=""):
2
iftitle:
3
returntitle+" "+first+" "+last
4
returnfirst+" "+last
5
6
# Positional arguments - order determines matching
# Skip middle parameter, specify just the one you need
16
print(format_name("Alan","Turing",title="Dr."))
>>>Output
Ada Lovelace
Marie Curie
Admiral Grace Hopper
Dr. Alan Turing
Notice in the second example how the order of keyword arguments does not matter. Python matches them by name, not position. This flexibility is especially valuable when a function has many optional parameters with defaults.
Why Use Keyword Arguments?
Consider a function call like calculate(100, 0.05, True, False). Without context, you cannot tell what these values mean. Keyword arguments make the code self-documenting:
•Hard to Understand
calculate(100, 0.05, True, False)
What do these arguments mean?
Easy to mix up the order
Must look at function definition
•Self-Documenting
calculate(
amount=100,
rate=0.05,
compound=True,
round_result=False)
The keyword argument version reads like documentation. Anyone reviewing the code immediately understands what each value represents without checking the function definition.
Optional Parameter Skipping
When a function has multiple optional parameters with defaults, keyword arguments let you skip parameters you do not need. This is impossible with positional arguments alone:
Without keyword arguments, you would need to provide values for author, date, and draft just to change format. Keyword arguments let you specify exactly what you need.
Keyword-Only Arguments
You can require certain arguments to be passed only by name using * in the parameter list. Everything after the * must be provided as a keyword argument:
Connecting to api.example.com:443 [SECURE] timeout=30 retries=3
Connecting to database.local:5432 timeout=60 retries=5
This design prevents confusion. Without keyword-only arguments, calling connect("host", 443, True, 30, 3) leaves readers guessing what True, 30, and 3 represent. Requiring keywords makes every call self-documenting.
TIP
Use keyword-only arguments when a function has boolean flags or multiple optional parameters of the same type. This prevents bugs from accidentally swapped arguments.
Test your understanding of keyword arguments by choosing the right syntax to call a function with named parameters.
Fill in the Blank
> A greet function has parameters name, greeting="Hi", and punctuation="!". You want to change the greeting to "Welcome" while keeping the default punctuation. Pick the right calling syntax.
def greet(name, greeting="Hi", punctuation="!"):
return greeting + ", " + name + punctuation
result = greet("Alice", )
print(result)
Positional and Keyword Mix
When mixing positional and keyword arguments in a call, positional arguments must come first. Once you use a keyword argument, all remaining arguments must also be keywords:
1
defexample(a,b,c,d):
2
print(a,b,c,d)
3
4
# Valid combinations:
5
example(1,2,3,4)
6
example(1,2,c=3,d=4)
7
example(1,b=2,c=3,d=4)
8
example(a=1,b=2,c=3,d=4)
9
10
# Invalid - positional cannot follow keyword:
11
# example(a=1, 2, 3, 4) # SyntaxError
12
# example(1, b=2, 3, d=4) # SyntaxError
>>>Output
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
Multiple Return Values
Daily Life
Interviews
Return multiple values from functions
Functions often need to produce multiple related results. Python makes this natural by allowing functions to return multiple values at once. Python packages them into a tuple, which you can unpack into separate variables. This pattern is common throughout Python's standard library and professional codebases.
How Multiple Returns Work
Write return a, b, c to return multiple values
Python automatically creates a tuple (a, b, c)
The caller can unpack into separate variables or keep as a tuple
Returning Multiple Values
Separate return values with commas. Python automatically creates a tuple:
1
defget_min_max(numbers):
2
"""Find min and max."""
3
smallest=numbers[0]
4
largest=numbers[0]
5
fornuminnumbers:
6
ifnum<smallest:
7
smallest=num
8
ifnum>largest:
9
largest=num
10
returnsmallest,largest
11
12
data=[5,2,8,1,9,3,7]
13
14
# Unpack into two variables
15
minimum,maximum=get_min_max(data)
16
print("Minimum:",minimum)
17
print("Maximum:",maximum)
18
19
# Or keep as a tuple
20
result=get_min_max(data)
21
print("As tuple:",result)
22
print("Access first:",result[0])
>>>Output
Minimum: 1
Maximum: 9
As tuple: (1, 9)
Access first: 1
The return statement creates a tuple. You can unpack it immediately into separate variables (most common) or store the tuple for later access. Both approaches are valid; choose based on how you will use the values.
01
Comma Separation
Write return a, b to return multiple values
02
Automatic Tuple
Python wraps the values into a tuple behind the scenes
03
Unpacking
Use x, y = func() to split the tuple into variables
04
Partial Unpack
Use _ to skip values you do not need
Multiple Return Patterns
Multiple return values are especially useful when computing related results that naturally belong together:
1
defdivide_with_remainder(dividend,divisor):
2
"""Return quotient and remainder."""
3
quotient=dividend//divisor
4
remainder=dividend%divisor
5
returnquotient,remainder
6
7
defget_dimensions(rectangle):
8
"""Extract width and height."""
9
returnrectangle["width"],rectangle["height"]
10
11
defanalyze_scores(scores):
12
"""Return count, sum, avg."""
13
count=len(scores)
14
total=sum(scores)
15
average=total/countifcount>0else0
16
returncount,total,average
17
18
# Using the functions
19
q,r=divide_with_remainder(17,5)
20
print(f"17 / 5 = {q} remainder {r}")
21
22
rect={"width":10,"height":5}
23
w,h=get_dimensions(rect)
24
print(f"Width: {w}, Height: {h}, Area: {w * h}")
25
26
grades=[85,90,78,92,88]
27
n,s,avg=analyze_scores(grades)
28
print(f"Count: {n}, Sum: {s}, Average: {avg}")
>>>Output
17 / 5 = 3 remainder 2
Width: 10, Height: 5, Area: 50
Count: 5, Sum: 433, Average: 86.6
Ignoring Some Return Values
Sometimes you only need some of the returned values. Use _ as a placeholder for values you want to ignore:
1
defget_user_info():
2
"""Return name, age, email, and role."""
3
return"alice",30,"alice@example.com","admin"
4
5
# Only care about name and email
6
name,_,email,_=get_user_info()
7
print("Name:",name)
8
print("Email:",email)
9
10
# Only care about the first value
11
first,*rest=get_user_info()
12
print("First:",first)
13
print("Rest:",rest)
>>>Output
Name: alice
Email: alice@example.com
First: alice
Rest: [30, 'alice@example.com', 'admin']
The _ is a valid variable name but signals "I am intentionally ignoring this value." The *rest syntax collects remaining values into a list.
TIP
If a function returns more than 3-4 values, consider returning a dictionary or a named tuple instead. Too many values become hard to track and easy to mix up.
Success Status with Return
A common pattern is returning a success indicator alongside the actual result. This lets callers easily check if the operation succeeded:
1
defsafe_divide(a,b):
2
"""Divide a by b with status."""
3
ifb==0:
4
returnFalse,0
5
returnTrue,a/b
6
7
defprocess_division(x,y):
8
success,result=safe_divide(x,y)
9
ifsuccess:
10
print(f"{x} / {y} = {result}")
11
else:
12
print(f"Cannot divide {x} by {y}")
13
14
process_division(10,2)
15
process_division(10,0)
16
process_division(15,3)
>>>Output
10 / 2 = 5.0
Cannot divide 10 by 0
15 / 3 = 5.0
This pattern avoids exceptions for expected failure cases and makes error handling explicit in the code structure.
Python Quiz
> A function returns three values: total, average, and count. Pick the correct built-in to compute the total of all numbers, and the built-in that counts how many elements are in the list.
Returning multiple values as a tuple is a Python idiom that keeps related results together without the overhead of a class or dictionary. The caller can unpack them into named variables for clarity.
The underscore convention for ignored values is widely recognized in Python. Using _ signals to other developers that the value is intentionally discarded, making the code self-documenting.
When a function produces more than three or four related values, consider returning a named tuple or dataclass instead. Named fields make it impossible to confuse the order of unpacked values.
Nested Functions
Daily Life
Interviews
Encapsulate helpers with nesting
You can define functions inside other functions. The inner function, called a nested function, is only accessible within the outer function. This keeps helper logic private and encapsulated, preventing pollution of the broader namespace.
Why Nest Functions?
Hide implementation details only relevant to the containing function
Keep related code together and avoid namespace pollution
Access variables from the enclosing function's scope
Basic Nested Function
Define helper functions inside the main function when they are only meaningful in that context:
1
defformat_greeting(name,formal=False):
2
"""Format a greeting."""
3
4
defadd_title(n):
5
"""Add formal title."""
6
ifformal:
7
return"Dr. "+n
8
returnn
9
10
defadd_punctuation(text):
11
"""Add ending."""
12
ifformal:
13
returntext+"."
14
returntext+"!"
15
16
formatted_name=add_title(name)
17
message="Hello, "+formatted_name
18
returnadd_punctuation(message)
19
20
print(format_greeting("Smith",formal=True))
21
print(format_greeting("Alice"))
22
23
# add_title("Bob") # NameError
>>>Output
Hello, Dr. Smith.
Hello, Alice!
The functions add_title and add_punctuation exist only inside format_greeting. They are helpers that only make sense in that context. Notice how they can access formal from the enclosing function.
Organizing Complex Logic
Nested functions help organize complex functions into clear, named steps without creating module-level functions that are only used once:
1
defcalc_total(items,tax=0.08,code=None):
2
"""Calculate total with tax and optional discount."""
Each step of the calculation has a clear, descriptive name. The main flow at the bottom reads like documentation: get subtotal, apply discount, add tax. The helper functions encapsulate the details.
Benefits of Nested Functions
Encapsulation: Hide helpers that are only used internally
Organization: Keep related code physically together
Scope access: Inner functions can use outer variables
Readability: Break complex logic into named pieces
Python Quiz
> A nested helper function accesses the threshold from its enclosing scope. Pick the built-in that keeps only items passing the test, and the one that counts how many survived.
Nested functions excel at keeping helper logic close to the code that uses it. When a function is only meaningful inside one context, defining it there prevents it from cluttering the module namespace.
Inner functions can read variables from their enclosing function without any special declaration. This makes them ideal for helpers that need to share configuration or state with the outer function.
The single-responsibility principle applies at every level. Even nested helper functions should do one thing. When a nested function starts doing two things, consider splitting it into two separate helpers or promoting it to a module-level function.
Closures
Daily Life
Interviews
Build function factories with closures
A closure is a function that remembers variables from the scope where it was created, even after that scope has finished executing. This powerful pattern lets you create customized functions and maintain state between calls without using global variables or classes.
What Makes a Closure?
An inner function references variables from an enclosing function
That inner function is returned or passed elsewhere
The inner function closes over the enclosing variables, keeping them alive
Creating a Closure
When a nested function references a variable from its enclosing function and is returned, Python creates a closure that captures that variable:
1
defmake_multiplier(factor):
2
"""Create a function that multiplies by factor."""
3
4
defmultiply(x):
5
# Uses factor from enclosing scope
6
returnx*factor
7
8
returnmultiply
9
10
# Create specialized multiplier functions
11
double=make_multiplier(2)
12
triple=make_multiplier(3)
13
times_ten=make_multiplier(10)
14
15
# Each function remembers its factor
16
print("double(5):",double(5))
17
print("triple(5):",triple(5))
18
print("times_ten(5):",times_ten(5))
19
20
# They are independent
21
print("double(100):",double(100))
22
print("triple(100):",triple(100))
>>>Output
double(5): 10
triple(5): 15
times_ten(5): 50
double(100): 200
triple(100): 300
Each call to make_multiplier creates a new multiply function that remembers the factor value from when it was created. Even though make_multiplier has finished executing, the returned function still has access to factor.
How Closures Work
When you call make_multiplier(2), Python creates a new multiply function with factor=2 attached. This attachment is the closure. When you later call double(5), the multiply function looks up factor in its closure and finds 2.
Closure Mechanics
Inner function references variable from enclosing scope
Python captures the variable itself, not just its current value
Captured variables are stored in the function's closure
Each call to outer function creates a new closure
Closures can share captured variables with other closures
Function Factory Pattern
Closures enable the function factory pattern, where one function creates and returns customized versions of another function:
1
defmake_formatter(prefix,suffix):
2
"""Create a custom formatter."""
3
4
defformat(text):
5
returnprefix+text+suffix
6
7
returnformat
8
9
# Create specialized formatters
10
make_bold=make_formatter("**","**")
11
make_italic=make_formatter("_","_")
12
make_code=make_formatter("`","`")
13
make_heading=make_formatter("# "," #")
14
15
# Use them
16
print(make_bold("important"))
17
print(make_italic("emphasis"))
18
print(make_code("variable"))
19
print(make_heading("Title"))
>>>Output
**important**
_emphasis_
`variable`
# Title #
Each formatter remembers its prefix and suffix. This is more flexible than writing four separate functions and more organized than using global variables.
Closures with Mutable State
Closures can maintain state across multiple calls. This is useful for counters, accumulators, and other stateful behavior:
1
defmake_counter(start=0):
2
"""Increment counter."""
3
count=start
4
5
defcounter():
6
nonlocalcount
7
count=count+1
8
returncount
9
10
returncounter
11
12
# Two independent counters
13
counter_a=make_counter()
14
counter_b=make_counter(100)
15
16
print("A:",counter_a())
17
print("A:",counter_a())
18
print("B:",counter_b())
19
print("A:",counter_a())
20
print("B:",counter_b())
>>>Output
A: 1
A: 2
B: 101
A: 3
B: 102
The nonlocal keyword tells Python to modify the count variable from the enclosing scope, rather than creating a new local variable. Without nonlocal, the assignment count = count + 1 would create a new local variable.
TIP
Use nonlocal when you need to modify an enclosing variable, not just read it. Reading works automatically; writing requires explicit declaration.
Configurable Validators
Closures are perfect for creating configurable validation functions:
Closures that maintain mutable state require careful use of the nonlocal keyword. See if you can spot the bug in this counter closure.
Debug Challenge
> This closure tries to maintain a mutable counter, but assigning to count inside the inner function creates a new local variable instead of updating the enclosing one.
UnboundLocalError: cannot access local variable "count" where it is not associated with a value
Always declare nonlocal when modifying enclosing variables
Name inner functions descriptively
Keep closures focused on a single responsibility
✗Don't
Use closures when a simple class would be clearer
Forget nonlocal when assigning to outer variables
Create deeply nested closures that are hard to follow
Rely on closures for complex state with many variables
The nonlocal keyword is the key distinction between reading and writing enclosing variables. Reading works automatically, but writing requires an explicit nonlocal declaration so Python knows not to create a new local variable.
Closures provide an alternative to classes when you only need state associated with a single behavior. A factory that returns a configured closure is lighter-weight than a class with __init__ and a single method.
Docstrings
Daily Life
Interviews
Document functions with docstrings
A docstring is a string literal that appears as the first statement in a function, class, or module. It documents what the code does, what parameters it expects, and what it returns. Unlike comments, docstrings are accessible at runtime and used by tools like help() and documentation generators.
Basic Docstrings
Use triple quotes for docstrings. A simple docstring is a single line that describes what the function does:
1
defcalculate_area(width,height):
2
"""Calculate the area of a rectangle."""
3
returnwidth*height
4
5
defis_prime(n):
6
"""Check if n is a prime number."""
7
ifn<2:
8
returnFalse
9
foriinrange(2,int(n**0.5)+1):
10
ifn%i==0:
11
returnFalse
12
returnTrue
13
14
# Access docstrings at runtime
15
print("calculate_area:",calculate_area.__doc__)
16
print("is_prime:",is_prime.__doc__)
17
18
# Use the help function
19
# help(calculate_area) # Shows docstring
>>>Output
calculate_area: Calculate the area of a rectangle.
is_prime: Check if n is a prime number.
The docstring becomes the function's __doc__ attribute. IDEs, the help() function, and documentation generators all use this attribute.
Multi-line Docstrings
For functions with parameters, return values, or complex behavior, use a multi-line docstring that documents the interface completely:
"""
Calculate the discounted price with a minimum floor.
Applies the specified percentage discount to the price,
ensuring the result does not fall below the minimum price.
Args:
price: The original price as a positive number.
discount_percent: Discount as a percentage (0-100).
min_price: Minimum allowed price (default: 0).
Returns:
The discounted price, not less than min_price.
Raises:
ValueError: If price < 0 or bad discount.
Example:
>>> calculate_discount(100, 20)
80.0
>>> calculate_discount(100, 90, min_price=20)
20
"""
3
ifprice<0:
4
raiseValueError("Price cannot be negative")
5
ifnot0<=discount_percent<=100:
6
raiseValueError("Discount must be between 0 and 100")
7
8
discounted=price*(1-discount_percent/100)
9
returnmax(discounted,min_price)
This multiline docstring format provides everything a user needs: a summary, detailed description, parameter documentation, return value, exceptions, and usage examples.
Docstring Structure
A well-structured docstring contains several key components that make it comprehensive and useful.
Components of a Good Docstring
Summary line: Brief description ending with a period (first line)
Blank line: Separates summary from detailed description
Detailed description: Explains behavior, context, side effects
Args section: Documents each parameter with type and description
Returns section: Describes return value(s) and their types
Raises section: Lists exceptions that might be raised
Example section: Shows usage with expected output
Docstring Best Practices
The difference between a helpful docstring and a useless one comes down to specificity and completeness.
•Poor Docstring
"""Does stuff."""
Too vague - what stuff?
No parameter documentation
No return value info
•Good Docstring
"""Calculate monthly payment.
Args:
principal: Loan amount in dollars
rate: Annual interest rate"""
Write docstrings for any function that will be used by others or that you might return to later. The summary line should be specific enough that someone can understand the function's purpose without reading the implementation.
When to Write Docstrings
Docstring Guidelines
Public functions: Always document thoroughly
Private helpers: Brief docstring if logic is complex
Library code: Comprehensive docs with examples
Internal scripts: At minimum, describe what the function does
Simple one-liners: Docstring optional if purpose is obvious
The most valuable docstrings go beyond restating the obvious.
TIP
Write docstrings that explain WHY and HOW, not just WHAT. "Sorts the list" is obvious from the name; "Uses quicksort for O(n log n) average performance" adds value.
Consistent docstring formatting also unlocks powerful tooling.
❯❯❯PUTTING IT ALL TOGETHER
> You are a data engineer at Palantir building a configurable data validation library where each check function is self-documenting, returns both a pass/fail flag and a diagnostic message, uses internal helper logic, and carries its configuration in a captured environment.
Keyword arguments make each validation function call self-documenting so reviewers immediately see which threshold or field name each argument targets.
return values let every check function return both a boolean pass/fail and a human-readable diagnostic string in a single tuple without a wrapper class.
Nested functions define private helper logic inside each validator so implementation details are hidden from callers and cannot be called elsewhere in the library.
Closures capture the threshold or rule configuration at definition time so each returned validator carries its own settings without requiring global state.
KEY TAKEAWAYS
Keyword arguments (name=value) make function calls self-documenting
Use * in the parameter list to require keyword-only arguments
Multiple return values are packed as tuples: return x, y, z
Use _ to ignore unwanted return values
Nested functions are private to their enclosing function
Closures capture and remember enclosing scope variables
Use nonlocal to modify captured variables
Docstrings document functions and are accessible via __doc__
Good docstrings include Args, Returns, and Examples sections
When calling a function, you can specify arguments by name instead of relying on position. This makes code clearer, especially for functions with many parameters or when the meaning of arguments is not obvious. Keyword arguments also let you skip optional parameters and provide only the ones you need. Positional vs Keyword Args You have been using positional arguments exclusively until now, where the order of values determines which parameter receives which value. Keyword arguments let you expli
Functions often need to produce multiple related results. Python makes this natural by allowing functions to return multiple values at once. Python packages them into a tuple, which you can unpack into separate variables. This pattern is common throughout Python's standard library and professional codebases. Returning Multiple Values The return statement creates a tuple. You can unpack it immediately into separate variables (most common) or store the tuple for later access. Both approaches are v
You can define functions inside other functions. The inner function, called a nested function, is only accessible within the outer function. This keeps helper logic private and encapsulated, preventing pollution of the broader namespace. Basic Nested Function Define helper functions inside the main function when they are only meaningful in that context: Organizing Complex Logic Nested functions help organize complex functions into clear, named steps without creating module-level functions that a
A closure is a function that remembers variables from the scope where it was created, even after that scope has finished executing. This powerful pattern lets you create customized functions and maintain state between calls without using global variables or classes. Creating a Closure When a nested function references a variable from its enclosing function and is returned, Python creates a closure that captures that variable: How Closures Work When you call make_multiplier(2), Python creates a n
Basic Docstrings Use triple quotes for docstrings. A simple docstring is a single line that describes what the function does: Multi-line Docstrings For functions with parameters, return values, or complex behavior, use a multi-line docstring that documents the interface completely: This multiline docstring format provides everything a user needs: a summary, detailed description, parameter documentation, return value, exceptions, and usage examples. Docstring Structure A well-structured docstring