Pinterest's backend reduces database load by over 90% on hot endpoints by wrapping expensive functions with caching decorators that store results and return them instantly on repeat calls. A single @lru_cache decorator can turn a function that queries a database on every call into one that queries it once and serves the cached result for thousands of subsequent requests. The advanced functional patterns in this lesson, including decorators, memoization, and generator-based pipelines, are what power Pinterest's infrastructure serving 450 million users each month.
Lambda Functions
Daily Life
Interviews
Write concise inline lambda functions
Sometimes you need a simple function for a single purpose, and defining it with def feels like overkill. Python provides lambda functions for exactly this situation. A lambda is an anonymous function defined in a single expression. Lambdas are perfect for short, throwaway functions that you'll only use once.
The term "lambda" comes from lambda calculus, a mathematical system for expressing computation. Do not let the fancy name intimidate you. lambda functions are simply a concise way to write small functions inline.
Lambda Syntax
A lambda has the form lambda arguments: expression. Unlike regular functions, lambdas do not use the return keyword. They automatically return the result of their single expression:
1
# Regular function to add two numbers
2
defadd(x,y):
3
returnx+y
4
5
# Equivalent lambda expression
6
add_lambda=lambdax,y:x+y
7
8
# Both work exactly the same
9
print("Regular function:",add(3,5))
10
print("Lambda function:",add_lambda(3,5))
11
12
# Lambda with one argument
13
square=lambdax:x*x
14
print("Square of 4:",square(4))
15
16
# Lambda with no arguments
17
get_message=lambda:"Hello from lambda!"
18
print(get_message())
>>>Output
Regular function: 8
Lambda function: 8
Square of 4: 16
Hello from lambda!
Notice how the lambda version is more compact. We go from three lines to one. But this comes with a limitation: lambdas can only contain a single expression. They cannot have multiple statements, loops, or complex logic.
Lambda: Can and Cannot Do
Understanding lambda limitations helps you choose when to use them. The single-expression rule is strict but makes lambdas predictable and easy to read:
1
# Lambdas CAN use conditional expressions
2
max_value=lambdaa,b:aifa>belseb
3
print("Max of 10 and 7:",max_value(10,7))
4
5
# Lambdas CAN use multiple operations in one expression
The best use cases for lambdas are situations where you need a simple function for immediate, one-time use. Here's how to decide:
•Good for Lambdas
Simple one-line operations
Immediate, one-time use
Passing to higher-order functions
Quick transformations and filters
Callback functions
•Use Named Functions
Complex logic or conditions
Reused in multiple places
Needs documentation or tests
Multiple statements needed
Debugging is important
Higher-Order Lambdas
Lambdas shine brightest when used with functions that take other functions as arguments. These are called higher-order functions. You'll see this pattern constantly in data processing:
1
defapply_operation(a,b,operation):
2
"""Apply operation to a and b."""
3
returnoperation(a,b)
4
5
result1=apply_operation(10,3,lambdax,y:x+y)
6
result2=apply_operation(10,3,lambdax,y:x-y)
7
result3=apply_operation(10,3,lambdax,y:x*y)
8
result4=apply_operation(10,3,lambdax,y:x**y)
9
result5=apply_operation(10,3,lambdax,y:x//y)
10
11
print("10 + 3 =",result1)
12
print("10 - 3 =",result2)
13
print("10 * 3 =",result3)
14
print("10 ^ 3 =",result4)
15
print("10 // 3 =",result5)
>>>Output
10 + 3 = 13
10 - 3 = 7
10 * 3 = 30
10 ^ 3 = 1000
10 // 3 = 3
Each lambda defines a different operation inline. This is much more concise than defining five separate named functions that you would only use once.
Lambdas for Sorting
One of the most common uses for lambdas is with sorted() or list's sort() method. The key parameter accepts a function that extracts a comparison value from each element:
1
# List of (name, age) tuples
2
people=[("Alice",30),("Bob",25),("Charlie",35)]
3
4
# Sort by name (first element) - alphabetically
5
by_name=sorted(people,key=lambdaperson:person[0])
6
print("By name:",by_name)
7
8
# Sort by age (second element) - numerically
9
by_age=sorted(people,key=lambdaperson:person[1])
10
print("By age:",by_age)
11
12
# Sort strings by length
13
words=["apple","pie","blueberry","cake"]
14
by_length=sorted(words,key=lambdaword:len(word))
15
print("By length:",by_length)
16
17
# Sort by absolute value
18
numbers=[-5,3,-1,7,-3]
19
by_abs=sorted(numbers,key=lambdax:abs(x))
20
print("By absolute value:",by_abs)
>>>Output
By name: [('Alice', 30), ('Bob', 25), ('Charlie', 35)]
By age: [('Bob', 25), ('Alice', 30), ('Charlie', 35)]
By length: ['pie', 'cake', 'apple', 'blueberry']
By absolute value: [-1, 3, -3, -5, 7]
The lambda tells Python how to compare elements. Without it, Python would compare tuples by their first element then second element. The lambda lets you control exactly what gets compared.
TIP
If a lambda gets complex or hard to read, convert it to a named function. Code clarity is more important than brevity. Your future self will thank you.
Practice choosing the right lambda expression for a sorting operation. Pay attention to which element the key function extracts.
Fill in the Blank
> You have a list of (name, salary) tuples and need to sort them with the highest-paid employee first. Pick the lambda that extracts the right element for sorting.
employees = [("Alice", 75000), ("Bob", 95000), ("Carol", 60000)]
result = sorted(employees, key=, reverse=True)
for name, sal in result:
print(name, sal)
Common Lambda Pitfalls
There are some common mistakes to avoid when using lambda functions:
1
# PITFALL 1: Lambda assigned to variable
2
# This works but is considered poor style:
3
multiply=lambdax,y:x*y
4
5
# Better - use def for named functions:
6
defmultiply_better(x,y):
7
returnx*y
8
9
# PITFALL 2: Lambdas in a loop capture by reference
10
# Creates 3 functions using FINAL value of i
11
funcs=[]
12
foriinrange(3):
13
# All will return 2!
14
funcs.append(lambda:i)
15
16
# See the problem:
17
print("Broken:",[f()forfinfuncs])
18
19
# FIX: Use default argument to capture current value
20
funcs_fixed=[]
21
foriinrange(3):
22
# Captures each value
23
funcs_fixed.append(lambdai=i:i)
24
25
print("Fixed:",[f()forfinfuncs_fixed])
>>>Output
Broken: [2, 2, 2]
Fixed: [0, 1, 2]
The loop pitfall catches many programmers. The lambda captures a reference to i, not its current value. Using a default argument i=i forces Python to copy the current value.
Functions as Objects
Daily Life
Interviews
Pass and store functions as values
In Python, functions are objects like any other value. You can store them in variables, pass them to other functions, return them from functions, and even store them in data structures. This concept is called "first-class functions."
This idea might seem abstract at first, but it's fundamental to Python's flexibility. Understanding it unlocks powerful patterns like callbacks, strategies, and the decorator pattern we'll explore later.
Functions Are Values
A function name without parentheses refers to the function object itself, not the result of calling it. This distinction is crucial:
1
defgreet(name):
2
return"Hello, "+name+"!"
3
4
# greet - the function object
5
# greet("Alice") - calls the function
6
7
# Store function in a variable
8
say_hello=greet
9
10
# Now both names refer to the same function
11
print(greet("Alice"))
12
print(say_hello("Bob"))
13
14
# Prove they're the same
15
print("Same function?",greetissay_hello)
16
17
# Functions are objects with attributes
18
print("Function name:",greet.__name__)
19
print("Type:",type(greet))
>>>Output
Hello, Alice!
Hello, Bob!
Same function? True
Function name: greet
Type: <class 'function'>
Notice the difference: greet is the function object, while greet("Alice") calls the function and returns its result ("Hello, Alice!"). Adding parentheses changes everything.
Functions in Collections
Since functions are objects, you can store them in lists, dictionaries, or any other data structure. This enables powerful patterns:
1
defadd(a,b):
2
returna+b
3
4
defsubtract(a,b):
5
returna-b
6
7
defmultiply(a,b):
8
returna*b
9
10
operations={"+":add,"-":subtract,"*":multiply}
11
12
print("10 + 5 =",operations["+"](10,5))
13
print("10 * 5 =",operations["*"](10,5))
14
15
forsymbol,funcinoperations.items():
16
print(f"10 {symbol} 5 = {func(10, 5)}")
>>>Output
10 + 5 = 15
10 * 5 = 50
10 + 5 = 15
10 - 5 = 5
10 * 5 = 50
This pattern is sometimes called a "dispatch table" or "strategy pattern". Instead of long if/elif chains, you look up the right function and call it. This makes code easier to extend.
Functions as Arguments
Functions that accept other functions as parameters are called higher-order functions. They're incredibly powerful because they let you customize behavior without changing code:
1
defapply_twice(func,value):
2
"""Apply func to value, then apply func to the result."""
The apply_twice function does not know what operation will be performed. It just calls whatever function is passed to it. This is the essence of higher-order programming.
Function Factories
Functions can create and return new functions. Combined with closures (from the intermediate lesson), this creates function factories:
1
defmake_power_function(exponent):
2
"""Raise to exponent."""
3
defpower(base):
4
returnbase**exponent
5
returnpower
6
7
# Create specialized power functions
8
square=make_power_function(2)
9
cube=make_power_function(3)
10
fourth=make_power_function(4)
11
12
# Each remembers its exponent
13
print("5 squared:",square(5))
14
print("5 cubed:",cube(5))
15
print("5 to the fourth:",fourth(5))
16
17
# Create a multiplier factory
18
defmake_multiplier(factor):
19
returnlambdax:x*factor
20
21
double=make_multiplier(2)
22
triple=make_multiplier(3)
23
24
print("Double 10:",double(10))
25
print("Triple 10:",triple(10))
>>>Output
5 squared: 25
5 cubed: 125
5 to the fourth: 625
Double 10: 20
Triple 10: 30
Each returned function captures its configuration value. The square function always uses exponent 2, cube uses 3. This is closure in action - the inner function remembers the outer function's variables.
Callback Pattern Example
Callbacks are functions passed to other code to be called later. This pattern is everywhere in programming:
> Use a higher-order function to apply a transformation to every element. Pick the built-in that applies a function to each item, and the built-in that converts the result into a usable list.
Treating functions as first-class values unlocks patterns that are fundamental to data engineering. Dispatch tables, callbacks, and function factories all rely on this core idea.
When you store functions in a dictionary, adding a new operation requires only a single dictionary entry. The code that calls the operation never changes, making your programs open for extension without requiring modification.
The map() and filter() built-ins are direct expressions of higher-order programming. Once you are comfortable with functions as values, these patterns become natural building blocks for clean, expressive code.
Decorators
Daily Life
Interviews
Wrap functions with @ decorators
A decorator is a function that wraps another function, adding behavior without modifying the original. Decorators combine everything you've learned: higher-order functions, closures, and function objects.
Decorators are everywhere in professional Python. Web frameworks like Flask and Django use them for routing. Testing frameworks use them for setup. Data libraries use them for caching. Understanding decorators is essential.
The Decorator Pattern
A decorator takes a function, creates a wrapper that adds behavior, and returns the wrapper. Let's build one step by step:
1
defannounce(func):
2
"""Announces function execution."""
3
defwrapper(x):
4
print("About to run: "+func.__name__)
5
result=func(x)
6
print("Finished running: "+func.__name__)
7
returnresult
8
returnwrapper
9
10
defsquare(x):
11
returnx*x
12
13
# Manually apply the decorator
14
decorated_square=announce(square)
15
16
# Call the decorated version
17
print("Result:",decorated_square(5))
18
19
print()
20
21
# Original is unchanged
22
print("Original still works:",square(5))
>>>Output
About to run: square
Finished running: square
Result: 25
Original still works: 25
The announce decorator wraps square in a wrapper that prints messages before and after calling the original function. The wrapper is a closure that remembers func.
The @ Syntax
Python provides the @ syntax as shorthand for applying decorators. This is cleaner and makes the decoration obvious:
1
defannounce(func):
2
"""Announces function calls."""
3
defwrapper(x):
4
print("Running:",func.__name__)
5
result=func(x)
6
returnresult
7
returnwrapper
8
9
@announce
10
defdouble(x):
11
returnx*2
12
13
@announce
14
defadd_ten(x):
15
returnx+10
16
17
# Both are decorated
18
print("Double 5:",double(5))
19
print()
20
print("Add 10 to 5:",add_ten(5))
>>>Output
Running: double
Double 5: 10
Running: add_ten
Add 10 to 5: 15
Using @announce above a function definition is exactly equivalent to writing double = announce(double) after the definition. The @ syntax is just cleaner.
Practical Decorator: Timing
One of the most useful decorators measures how long a function takes to execute:
1
importtime
2
3
deftimer(func):
4
"""Decorator that times function execution."""
5
defwrapper(*args,**kwargs):
6
start=time.time()
7
result=func(*args,**kwargs)
8
end=time.time()
9
print(f"{func.__name__} took {end - start:.4f} seconds")
10
returnresult
11
returnwrapper
12
13
@timer
14
defslow_function():
15
time.sleep(1)
16
return"Done"
17
18
@timer
19
deffast_function():
20
returnsum(range(1000))
21
22
slow_function()
23
fast_function()
This pattern is extremely useful for performance profiling. Add the decorator to any function you want to measure, without changing the function itself.
Decorator with Validation
Decorators are perfect for validation because they separate the validation logic from the main function:
1
defrequire_positive(func):
2
"""Ensures first arg is positive."""
3
defwrapper(x):
4
ifx<=0:
5
return"Error: value must be positive"
6
returnfunc(x)
7
returnwrapper
8
9
@require_positive
10
defcalculate_square_root(x):
11
returnx**0.5
12
13
@require_positive
14
defcalculate_log(x):
15
# Simplified log calculation
16
returnstr(x)+" is positive, log would work"
17
18
print(calculate_square_root(16))
19
print(calculate_square_root(25))
20
print(calculate_square_root(-5))
21
print()
22
print(calculate_log(100))
23
print(calculate_log(-10))
>>>Output
4.0
5.0
Error: value must be positive
100 is positive, log would work
Error: value must be positive
The validation logic lives in one place - the decorator. Any function that needs this validation just adds the decorator. This is the DRY principle (Don't Repeat Yourself) in action.
Handling Any Arguments
Real-world decorators need to work with functions that have different signatures. Use *args and **kwargs to handle any combination of arguments:
1
deflog_call(func):
2
defwrapper(*args,**kwargs):
3
print(f"Call: {func.__name__}({args}, {kwargs})")
4
result=func(*args,**kwargs)
5
print(f"Return: {result}")
6
returnresult
7
returnwrapper
8
9
@log_call
10
defadd(a,b):
11
returna+b
12
13
@log_call
14
defgreet(name,msg="Hi"):
15
returnf"{msg}, {name}!"
16
17
add(3,5)
18
print()
19
greet("Alice",msg="Hello")
>>>Output
Call: add((3, 5), {})
Return: 8
Call: greet(('Alice',), {'msg': 'Hello'})
Return: Hello, Alice!
*args collects all positional arguments as a tuple. **kwargs collects all keyword arguments as a dictionary. The wrapper passes them along to the original function unchanged.
Stacking Decorators
You can apply multiple decorators to a single function. They apply from bottom to top:
One subtle but important detail makes decorators production-ready.
TIP
When writing decorators, use functools.wraps to preserve the original function's name and docstring. This helps with debugging and documentation.
A common decorator bug is forgetting to return the wrapper function. See if you can identify and fix the issue below.
Debug Challenge
> This decorator wraps a function to print a greeting, but the wrapped function's return value is lost. Fix it so the caller receives the original return value.
The decorated function prints "Hello!" but result is None instead of "I am Alice"
defadd_greeting(func):defwrapper(*args):print("Hello!")func(*args)returnwrapper@add_greetingdefsay_name(name):return"I am "+nameresult=say_name("Alice")print(result)
Decorators that swallow return values are one of the most common bugs in functional Python code. Always ensure the wrapper returns whatever the original function produces.
The @ syntax applies a decorator at definition time. Using @timer above a function is exactly equivalent to writing timer(my_function) after the definition. The shorthand keeps decoration visible and close to the function being decorated.
Decorators compose well with each other. Stacking @bold, @italic, and @uppercase above a single function applies each decorator in order from bottom to top, creating layered behavior without modifying any function's internal logic.
Recursion
Daily Life
Interviews
Solve nested problems with recursion
Recursion is when a function calls itself. This technique is elegant for problems that can be broken into smaller versions of the same problem. While it might seem strange at first, recursion is a natural fit for many algorithms.
Data engineers encounter recursion when traversing nested JSON, processing tree structures, working with file system hierarchies, or implementing divide-and-conquer algorithms. It's also a favorite topic in technical interviews.
The Two Parts of Recursion
Every recursive function must have two parts: a base case that stops the recursion, and a recursive case that calls itself with a smaller problem.
BaseStepTrust
Base
Base Case
When to stop recursing
Step
Recursive Case
Call self with smaller input
Trust
Leap of Faith
Assume recursive call works
1
defcountdown(n):
2
"""Count to 1."""
3
# Base case
4
ifn<=0:
5
print("Done!")
6
return
7
8
print(n)
9
10
# Recurse smaller
11
countdown(n-1)
12
13
countdown(5)
>>>Output
5
4
3
2
1
Done!
Each call to countdown passes a smaller number. Eventually n reaches 0, hitting the base case and stopping. Without the base case, the function would call itself forever (until Python stops it with an error).
Return Values in Recursion
Recursive functions can compute and return values. The classic example is factorial - the product of all positive integers up to n:
1
deffactorial(n):
2
"""Calculate n!."""
3
# Base case
4
ifn<=1:
5
return1
6
7
# n! = n * (n-1)!
8
returnn*factorial(n-1)
9
10
# Trace factorial(5):
11
# 5 * factorial(4)
12
# 5 * 4 * factorial(3)
13
# 5 * 4 * 3 * factorial(2)
14
# 5 * 4 * 3 * 2 * 1 = 120
15
16
print("5! =",factorial(5))
17
print("4! =",factorial(4))
18
print("3! =",factorial(3))
19
print("10! =",factorial(10))
>>>Output
5! = 120
4! = 24
3! = 6
10! = 3628800
factorial(5) calls factorial(4), which calls factorial(3), and so on. Each call waits for its recursive call to return before it can compute its result.
Thinking Recursively
To solve a problem recursively, ask yourself: "How can I express this in terms of a smaller version of the same problem?" Trust that the recursive call will work correctly.
1
defsum_to(n):
2
"""Sum of 1 + 2 + 3 + ... + n"""
3
# Base case: sum to 0 is 0
4
ifn<=0:
5
return0
6
# Recursive: sum to n = n + sum to (n-1)
7
returnn+sum_to(n-1)
8
9
defpower(base,exp):
10
"""Calculate base raised to exp power."""
11
# Base case: anything^0 = 1
12
ifexp==0:
13
return1
14
# Recursive: base^exp = base * base^(exp-1)
15
returnbase*power(base,exp-1)
16
17
defreverse_string(s):
18
"""Reverse a string recursively."""
19
# Base case: empty or single char
20
iflen(s)<=1:
21
returns
22
# Recursive: last char + reverse of rest
23
returns[-1]+reverse_string(s[:-1])
24
25
print("Sum 1 to 10:",sum_to(10))
26
print("2^8:",power(2,8))
27
print("Reverse 'hello':",reverse_string("hello"))
>>>Output
Sum 1 to 10: 55
2^8: 256
Reverse 'hello': olleh
Recursion Checklist
Identify the base case (when to stop)
Identify how to reduce to a smaller problem
Ensure each recursive call moves toward base case
Trust that the recursive call works correctly
Test with simple inputs first
Recursive Data Structures
Recursion is natural for nested data structures like lists within lists or tree-like JSON:
1
defsum_nested(data):
2
"""Sum all numbers in a nested list structure."""
3
total=0
4
foritemindata:
5
ifisinstance(item,list):
6
# Recursive case: sum the nested list
7
total+=sum_nested(item)
8
else:
9
# Base case: it's a number
10
total+=item
11
returntotal
12
13
# Nested list with multiple levels
14
nested=[1,[2,3],[4,[5,6]],7]
15
print("Sum of nested:",sum_nested(nested))
16
17
# Another example
18
deep=[[[1]],[[2]],[[3]]]
19
print("Sum of deep:",sum_nested(deep))
20
21
defflatten(nested_list):
22
"""Flatten a nested list into a single list."""
23
result=[]
24
foriteminnested_list:
25
ifisinstance(item,list):
26
result.extend(flatten(item))
27
else:
28
result.append(item)
29
returnresult
30
31
print("Flattened:",flatten([1,[2,[3,4]],5]))
>>>Output
Sum of nested: 28
Sum of deep: 6
Flattened: [1, 2, 3, 4, 5]
Processing nested JSON data in data engineering often uses exactly this pattern. The recursive function handles any depth of nesting automatically.
Recursion vs Iteration
Many problems can be solved with either recursion or loops. Each approach has trade-offs:
•Recursion
Elegant for trees and nested structures
Natural for divide-and-conquer
Matches mathematical definitions
Can be memory-intensive
Risk of stack overflow for deep recursion
•Iteration (loops)
More efficient for linear processing
Uses constant memory
No stack overflow risk
Can be harder to express some algorithms
Better for performance-critical code
1
# Same problem: sum 1 to n
2
3
defsum_recursive(n):
4
ifn<=0:
5
return0
6
returnn+sum_recursive(n-1)
7
8
defsum_iterative(n):
9
total=0
10
foriinrange(1,n+1):
11
total+=i
12
returntotal
13
14
print("Recursive sum to 100:",sum_recursive(100))
15
print("Iterative sum to 100:",sum_iterative(100))
16
17
# For simple cases, iteration is usually cleaner
18
# For nested structures, recursion is cleaner
>>>Output
Recursive sum to 100: 5050
Iterative sum to 100: 5050
TIP
Python has a default recursion limit of 1000 calls. For very deep recursion, you may need to increase it with sys.setrecursionlimit() or convert to iteration.
The Fibonacci Example
The Fibonacci sequence is a classic recursion example. Each number is the sum of the two before it: 0, 1, 1, 2, 3, 5, 8, 13...
1
deffibonacci(n):
2
"""Return the nth Fibonacci number."""
3
# Base cases: first two numbers
4
ifn<=0:
5
return0
6
ifn==1:
7
return1
8
9
# Recursive case: sum of two previous
10
returnfibonacci(n-1)+fibonacci(n-2)
11
12
# Print first 10 Fibonacci numbers
13
print("Fibonacci sequence:")
14
foriinrange(10):
15
print(f"fib({i}) = {fibonacci(i)}")
>>>Output
Fibonacci sequence:
fib(0) = 0
fib(1) = 1
fib(2) = 1
fib(3) = 2
fib(4) = 3
fib(5) = 5
fib(6) = 8
fib(7) = 13
fib(8) = 21
fib(9) = 34
Python Quiz
> A recursive function computes the factorial of n. Pick the comparison that defines the base case, and the arithmetic operator that combines n with the recursive result.
Recursion becomes natural with practice. Every recursive solution follows the same pattern: identify the simplest case that needs no further work, then express the general case in terms of a smaller version of the same problem.
Recursive functions for nested data structures handle arbitrary depth automatically. A flat loop cannot traverse data of unknown nesting depth, but a recursive function works regardless of how many layers deep the structure goes.
The trade-off between recursion and iteration is always worth considering. Recursion is expressive for tree-shaped problems, while iteration is more memory-efficient for linear sequences. Python's default recursion limit of 1000 (adjustable via sys.setrecursionlimit()) is a practical guide for when to switch approaches.
Function Composition
Daily Life
Interviews
Chain functions into data pipelines
Function composition means combining simple functions to build complex operations. The output of one function becomes the input of the next. This is a core concept in functional programming.
Data pipelines are perfect examples of composition. Raw data flows through a series of transformation functions: clean, validate, transform, aggregate, format. Each function does one thing well.
Manual Composition
The simplest form of composition is calling functions in sequence, where each function takes the previous result:
Each function takes text and returns modified text. We chain them together to build the final result. The functions are reusable - you can combine them in different ways.
Creating a Compose Function
We can create a higher-order function that composes two functions into one new function:
1
defcompose(f,g):
2
defcomposed(x):
3
returnf(g(x))
4
returncomposed
5
6
defdouble(x):
7
returnx*2
8
9
defadd_one(x):
10
returnx+1
11
12
defsquare(x):
13
returnx*x
14
15
double_after_add=compose(double,add_one)
16
print("double(add_one(5)):",double_after_add(5))
17
18
print("double then square:",compose(square,double)(3))
19
print("square then double:",compose(double,square)(3))
>>>Output
double(add_one(5)): 12
double then square: 36
square then double: 18
compose(f, g) creates a new function that first applies g then applies f to the result. Note the order: g runs first (inner), f runs second (outer).
Function Composition
You can compose any number of functions by chaining compose calls or creating a more flexible version:
1
defcompose_all(*functions):
2
defcomposed(x):
3
result=x
4
forfuncinreversed(functions):
5
result=func(result)
6
returnresult
7
returncomposed
8
9
defadd_one(x):
10
returnx+1
11
12
defdouble(x):
13
returnx*2
14
15
defsquare(x):
16
returnx*x
17
18
pipeline=compose_all(square,double,add_one)
19
print("Pipeline(2):",pipeline(2))
>>>Output
Pipeline(2): 36
Pipe: Left-to-Right Flow
Compose works right-to-left, which matches function nesting. But sometimes left-to-right is more intuitive. A pipe function applies functions in the order listed:
1
defpipe(*functions):
2
defpiped(x):
3
result=x
4
forfuncinfunctions:
5
result=func(result)
6
returnresult
7
returnpiped
8
9
defadd_one(x):
10
returnx+1
11
12
defdouble(x):
13
returnx*2
14
15
defto_string(x):
16
return"Result: "+str(x)
17
18
process=pipe(add_one,double,to_string)
19
print(process(5))
>>>Output
Result: 12
With pipe, the functions are listed in the order they execute. Many developers find this easier to read and reason about than right-to-left composition.
Real-World Data Pipeline
Composition shines in data processing. Each function handles one transformation, making the code modular and testable:
Each function in the pipeline does one thing well. You can test them individually, reuse them in other pipelines, and easily understand what each step does. This is the power of composition.
Partial Application
Partial application is a related concept: creating a new function by pre-filling some arguments. This makes composition more flexible:
1
defpartial(func,*fixed_args,**fixed_kwargs):
2
defwrapper(*args,**kwargs):
3
all_kwargs={**fixed_kwargs,**kwargs}
4
returnfunc(*fixed_args,*args,**all_kwargs)
5
returnwrapper
6
7
defmultiply(a,b):
8
returna*b
9
10
defgreet(message,name,punctuation="!"):
11
returnmessage+", "+name+punctuation
12
13
double=partial(multiply,2)
14
triple=partial(multiply,3)
15
say_hi=partial(greet,"Hi",punctuation="...")
16
17
print("double(5):",double(5))
18
print("triple(5):",triple(5))
19
print(say_hi("Bob"))
>>>Output
double(5): 10
triple(5): 15
Hi, Bob...
Python's standard library includes functools.partial for this purpose. It is commonly used to adapt functions for composition when their signatures do not quite match.
Common Mistakes
Even experienced developers make these mistakes with advanced functional programming:
Mistakes to Avoid
lambda used when a named function would be clearer
Forgetting the base case in recursion (infinite loop)
Not moving toward base case with each call (infinite loop)
Decorators that do not return the wrapper function
Decorator wrappers that do not pass *args and **kwargs
Deep recursion exceeding Python's stack limit
Complex lambda expressions that should be regular functions
1
# MISTAKE: Lambda that's too complex
2
bad=lambdax:xifx>0else(x*-1ifx<0else0)
3
# Better as a named function:
4
defabsolute_value(x):
5
ifx>0:
6
returnx
7
elifx<0:
8
returnx*-1
9
else:
10
return0
11
12
print("Lambda:",bad(-5))
13
print("Named:",absolute_value(-5))
14
15
# MISTAKE: Decorator not returning wrapper
16
defbroken_decorator(func):
17
defwrapper(*args):
18
print("Before")
19
result=func(*args)
20
print("After")
21
returnresult
22
# Forgot to return wrapper!
23
# This would make decorated functions None
24
25
# Correct:
26
defworking_decorator(func):
27
defwrapper(*args):
28
print("Decorated!")
29
returnfunc(*args)
30
returnwrapper
31
32
@working_decorator
33
defgreet():
34
return"Hello"
35
36
print(greet())
>>>Output
Lambda: 5
Named: 5
Decorated!
Hello
01
Lambdas
Concise anonymous functions for one-time inline use
02
Decorators
Wrap functions with reusable behavior via @ syntax
03
Recursion
Functions calling themselves to solve nested problems
04
Composition
Chain simple functions into powerful pipelines
These advanced functional patterns work together in real systems. Practice making architectural decisions that combine them effectively in the scenario below.
Data Pipeline ArchitectureStep 1
>
You are building a Python data pipeline that reads JSON logs from an API, transforms each record, validates fields, and writes clean data to a database. The pipeline must handle 100,000 records daily and be easy for your team to extend with new transformations.
raw_api_logs
timestamp
user_id
event
payload
2024-03-01T10:00:00
u_42
click
{"page": "home"}
2024-03-01T10:01:00
u_99
purchase
{"item": "widget", "amount": 29.99}
2024-03-01T10:02:00
u_42
click
{"page": "settings"}
May
2026
You need to apply several transformations in sequence: parse timestamps, validate user IDs, extract nested payload fields, and filter bot traffic. How do you structure the transformations?
Advanced functional techniques can dramatically simplify complex data transformations. Put your skills to the test with hands-on challenges in the Python Builder.
Function composition encourages you to write small, single-purpose functions and combine them. Each function is individually testable, and the composition is easy to read because each step has a descriptive name.
Partial application complements composition by letting you pre-configure a function's arguments. Combined with pipe or compose, partial application lets you build entire data transformation pipelines from reusable building blocks.
❯❯❯PUTTING IT ALL TOGETHER
> You are a senior data engineer at DoorDash building a modular ETL framework where transformation steps are defined as concise inline expressions, passed as arguments to orchestration functions, automatically timed and logged by a wrapping layer, and assembled into multi-stage pipelines through composition.
lambda functions define concise single-expression transformations for inline sorting keys and field extractors without creating named functions that clutter the module namespace.
Functions as first-class objects allow each transformation step to be stored in a list, passed as an argument, and dynamically swapped at runtime without changing the orchestration code.
Decorators wrap every ETL step function with automatic timing and error-logging behavior so instrumentation is applied uniformly without modifying any transformation's core logic.
Function composition chains simple single-purpose transformations into a multi-stage pipeline so each stage receives the output of the previous one in a readable left-to-right flow.
KEY TAKEAWAYS
lambda creates anonymous functions for simple, one-time operations - use them for sorting keys and callbacks
Functions are first-class objects: store in variables, pass as arguments, return from functions
Higher-order functions accept or return other functions, enabling powerful abstractions
Decorators wrap functions using the @ syntax - use them for logging, timing, validation, and caching
Every recursive function needs a base case and must move toward it with each call
Recursion is natural for nested data structures like trees and nested lists
Function composition chains simple functions to build complex data pipelines
Use *args and **kwargs in decorator wrappers to handle any function signature
Powerful patterns with functions
Category
Python
Difficulty
advanced
Duration
42 minutes
Challenges
0 hands-on challenges
Topics covered: Lambda Functions, Functions as Objects, Decorators, Recursion, Function Composition
Lambda Syntax Lambda: Can and Cannot Do Understanding lambda limitations helps you choose when to use them. The single-expression rule is strict but makes lambdas predictable and easy to read: When to Use Lambdas The best use cases for lambdas are situations where you need a simple function for immediate, one-time use. Here's how to decide: Higher-Order Lambdas Lambdas shine brightest when used with functions that take other functions as arguments. These are called higher-order functions. You'll
In Python, functions are objects like any other value. You can store them in variables, pass them to other functions, return them from functions, and even store them in data structures. This concept is called "first-class functions." This idea might seem abstract at first, but it's fundamental to Python's flexibility. Understanding it unlocks powerful patterns like callbacks, strategies, and the decorator pattern we'll explore later. Functions Are Values A function name without parentheses refer
A decorator is a function that wraps another function, adding behavior without modifying the original. Decorators combine everything you've learned: higher-order functions, closures, and function objects. Decorators are everywhere in professional Python. Web frameworks like Flask and Django use them for routing. Testing frameworks use them for setup. Data libraries use them for caching. Understanding decorators is essential. The Decorator Pattern A decorator takes a function, creates a wrapper t
Recursion is when a function calls itself. This technique is elegant for problems that can be broken into smaller versions of the same problem. While it might seem strange at first, recursion is a natural fit for many algorithms. Data engineers encounter recursion when traversing nested JSON, processing tree structures, working with file system hierarchies, or implementing divide-and-conquer algorithms. It's also a favorite topic in technical interviews. The Two Parts of Recursion Every recursiv
Function composition means combining simple functions to build complex operations. The output of one function becomes the input of the next. This is a core concept in functional programming. Data pipelines are perfect examples of composition. Raw data flows through a series of transformation functions: clean, validate, transform, aggregate, format. Each function does one thing well. Manual Composition The simplest form of composition is calling functions in sequence, where each function takes th