Functional Programming: Advanced

Pinterest's backend reduces database load by over 90% on hot endpoints by wrapping expensive functions with caching decorators that store results and return them instantly on repeat calls. A single @lru_cache decorator can turn a function that queries a database on every call into one that queries it once and serves the cached result for thousands of subsequent requests. The advanced functional patterns in this lesson, including decorators, memoization, and generator-based pipelines, are what power Pinterest's infrastructure serving 450 million users each month.

Lambda Functions

Daily Life
Interviews

Write concise inline lambda functions

Sometimes you need a simple function for a single purpose, and defining it with def feels like overkill. Python provides lambda functions for exactly this situation. A lambda is an anonymous function defined in a single expression. Lambdas are perfect for short, throwaway functions that you'll only use once.

The term "lambda" comes from lambda calculus, a mathematical system for expressing computation. Do not let the fancy name intimidate you. lambda functions are simply a concise way to write small functions inline.

Lambda Syntax

A lambda has the form lambda arguments: expression. Unlike regular functions, lambdas do not use the return keyword. They automatically return the result of their single expression:

1# Regular function to add two numbers
2def add(x, y):
3 return x + y
4
5# Equivalent lambda expression
6add_lambda = lambda x, y: x + y
7
8# Both work exactly the same
9print("Regular function:", add(3, 5))
10print("Lambda function:", add_lambda(3, 5))
11
12# Lambda with one argument
13square = lambda x: x * x
14print("Square of 4:", square(4))
15
16# Lambda with no arguments
17get_message = lambda: "Hello from lambda!"
18print(get_message())
>>>Output
Regular function: 8
Lambda function: 8
Square of 4: 16
Hello from lambda!

Notice how the lambda version is more compact. We go from three lines to one. But this comes with a limitation: lambdas can only contain a single expression. They cannot have multiple statements, loops, or complex logic.

Lambda: Can and Cannot Do

Understanding lambda limitations helps you choose when to use them. The single-expression rule is strict but makes lambdas predictable and easy to read:
1# Lambdas CAN use conditional expressions
2max_value = lambda a, b: a if a > b else b
3print("Max of 10 and 7:", max_value(10, 7))
4
5# Lambdas CAN use multiple operations in one expression
6hypotenuse = lambda a, b: (a**2 + b**2) ** 0.5
7print("Hypotenuse of 3,4:", hypotenuse(3, 4))
8
9# Lambdas CAN call other functions
10format_name = lambda first, last: first.upper() + " " + last.upper()
11print(format_name("jane", "doe"))
12
13# Lambdas CAN have default arguments
14greet = lambda name="World": "Hello, " + name + "!"
15print(greet())
16print(greet("Python"))
>>>Output
Max of 10 and 7: 10
Hypotenuse of 3,4: 5.0
JANE DOE
Hello, World!
Hello, Python!

When to Use Lambdas

The best use cases for lambdas are situations where you need a simple function for immediate, one-time use. Here's how to decide:
Good for Lambdas
  • Simple one-line operations
  • Immediate, one-time use
  • Passing to higher-order functions
  • Quick transformations and filters
  • Callback functions
Use Named Functions
  • Complex logic or conditions
  • Reused in multiple places
  • Needs documentation or tests
  • Multiple statements needed
  • Debugging is important

Higher-Order Lambdas

Lambdas shine brightest when used with functions that take other functions as arguments. These are called higher-order functions. You'll see this pattern constantly in data processing:
1def apply_operation(a, b, operation):
2 """Apply operation to a and b."""
3 return operation(a, b)
4
5result1 = apply_operation(10, 3, lambda x, y: x + y)
6result2 = apply_operation(10, 3, lambda x, y: x - y)
7result3 = apply_operation(10, 3, lambda x, y: x * y)
8result4 = apply_operation(10, 3, lambda x, y: x ** y)
9result5 = apply_operation(10, 3, lambda x, y: x // y)
10
11print("10 + 3 =", result1)
12print("10 - 3 =", result2)
13print("10 * 3 =", result3)
14print("10 ^ 3 =", result4)
15print("10 // 3 =", result5)
>>>Output
10 + 3 = 13
10 - 3 = 7
10 * 3 = 30
10 ^ 3 = 1000
10 // 3 = 3

Each lambda defines a different operation inline. This is much more concise than defining five separate named functions that you would only use once.

Lambdas for Sorting

One of the most common uses for lambdas is with sorted() or list's sort() method. The key parameter accepts a function that extracts a comparison value from each element:

1# List of (name, age) tuples
2people = [("Alice", 30), ("Bob", 25), ("Charlie", 35)]
3
4# Sort by name (first element) - alphabetically
5by_name = sorted(people, key=lambda person: person[0])
6print("By name:", by_name)
7
8# Sort by age (second element) - numerically
9by_age = sorted(people, key=lambda person: person[1])
10print("By age:", by_age)
11
12# Sort strings by length
13words = ["apple", "pie", "blueberry", "cake"]
14by_length = sorted(words, key=lambda word: len(word))
15print("By length:", by_length)
16
17# Sort by absolute value
18numbers = [-5, 3, -1, 7, -3]
19by_abs = sorted(numbers, key=lambda x: abs(x))
20print("By absolute value:", by_abs)
>>>Output
By name: [('Alice', 30), ('Bob', 25), ('Charlie', 35)]
By age: [('Bob', 25), ('Alice', 30), ('Charlie', 35)]
By length: ['pie', 'cake', 'apple', 'blueberry']
By absolute value: [-1, 3, -3, -5, 7]

The lambda tells Python how to compare elements. Without it, Python would compare tuples by their first element then second element. The lambda lets you control exactly what gets compared.

TIP
If a lambda gets complex or hard to read, convert it to a named function. Code clarity is more important than brevity. Your future self will thank you.
Practice choosing the right lambda expression for a sorting operation. Pay attention to which element the key function extracts.
Fill in the Blank

> You have a list of (name, salary) tuples and need to sort them with the highest-paid employee first. Pick the lambda that extracts the right element for sorting.

employees = [("Alice", 75000), ("Bob", 95000), ("Carol", 60000)]

result = sorted(employees, key=, reverse=True)
for name, sal in result:
    print(name, sal)

Common Lambda Pitfalls

There are some common mistakes to avoid when using lambda functions:
1# PITFALL 1: Lambda assigned to variable
2# This works but is considered poor style:
3multiply = lambda x, y: x * y
4
5# Better - use def for named functions:
6def multiply_better(x, y):
7 return x * y
8
9# PITFALL 2: Lambdas in a loop capture by reference
10# Creates 3 functions using FINAL value of i
11funcs = []
12for i in range(3):
13 # All will return 2!
14 funcs.append(lambda: i)
15
16# See the problem:
17print("Broken:", [f() for f in funcs])
18
19# FIX: Use default argument to capture current value
20funcs_fixed = []
21for i in range(3):
22 # Captures each value
23 funcs_fixed.append(lambda i=i: i)
24
25print("Fixed:", [f() for f in funcs_fixed])
>>>Output
Broken: [2, 2, 2]
Fixed: [0, 1, 2]

The loop pitfall catches many programmers. The lambda captures a reference to i, not its current value. Using a default argument i=i forces Python to copy the current value.

Functions as Objects

Daily Life
Interviews

Pass and store functions as values

In Python, functions are objects like any other value. You can store them in variables, pass them to other functions, return them from functions, and even store them in data structures. This concept is called "first-class functions."
This idea might seem abstract at first, but it's fundamental to Python's flexibility. Understanding it unlocks powerful patterns like callbacks, strategies, and the decorator pattern we'll explore later.

Functions Are Values

A function name without parentheses refers to the function object itself, not the result of calling it. This distinction is crucial:
1def greet(name):
2 return "Hello, " + name + "!"
3
4# greet - the function object
5# greet("Alice") - calls the function
6
7# Store function in a variable
8say_hello = greet
9
10# Now both names refer to the same function
11print(greet("Alice"))
12print(say_hello("Bob"))
13
14# Prove they're the same
15print("Same function?", greet is say_hello)
16
17# Functions are objects with attributes
18print("Function name:", greet.__name__)
19print("Type:", type(greet))
>>>Output
Hello, Alice!
Hello, Bob!
Same function? True
Function name: greet
Type: <class 'function'>

Notice the difference: greet is the function object, while greet("Alice") calls the function and returns its result ("Hello, Alice!"). Adding parentheses changes everything.

Functions in Collections

Since functions are objects, you can store them in lists, dictionaries, or any other data structure. This enables powerful patterns:
1def add(a, b):
2 return a + b
3
4def subtract(a, b):
5 return a - b
6
7def multiply(a, b):
8 return a * b
9
10operations = {"+": add, "-": subtract, "*": multiply}
11
12print("10 + 5 =", operations["+"](10, 5))
13print("10 * 5 =", operations["*"](10, 5))
14
15for symbol, func in operations.items():
16 print(f"10 {symbol} 5 = {func(10, 5)}")
>>>Output
10 + 5 = 15
10 * 5 = 50
10 + 5 = 15
10 - 5 = 5
10 * 5 = 50
This pattern is sometimes called a "dispatch table" or "strategy pattern". Instead of long if/elif chains, you look up the right function and call it. This makes code easier to extend.

Functions as Arguments

Functions that accept other functions as parameters are called higher-order functions. They're incredibly powerful because they let you customize behavior without changing code:
1def apply_twice(func, value):
2 """Apply func to value, then apply func to the result."""
3 return func(func(value))
4
5def add_one(x):
6 return x + 1
7
8def double(x):
9 return x * 2
10
11def square(x):
12 return x * x
13
14# Same higher-order function, different behaviors
15print("Add 1 twice to 5:", apply_twice(add_one, 5))
16print("Double twice 3:", apply_twice(double, 3))
17print("Square twice 2:", apply_twice(square, 2))
18
19# Works with lambdas too
20print("Add 10 twice:", apply_twice(lambda x: x + 10, 0))
>>>Output
Add 1 twice to 5: 7
Double twice 3: 12
Square twice 2: 16
Add 10 twice: 20

The apply_twice function does not know what operation will be performed. It just calls whatever function is passed to it. This is the essence of higher-order programming.

Function Factories

Functions can create and return new functions. Combined with closures (from the intermediate lesson), this creates function factories:
1def make_power_function(exponent):
2 """Raise to exponent."""
3 def power(base):
4 return base ** exponent
5 return power
6
7# Create specialized power functions
8square = make_power_function(2)
9cube = make_power_function(3)
10fourth = make_power_function(4)
11
12# Each remembers its exponent
13print("5 squared:", square(5))
14print("5 cubed:", cube(5))
15print("5 to the fourth:", fourth(5))
16
17# Create a multiplier factory
18def make_multiplier(factor):
19 return lambda x: x * factor
20
21double = make_multiplier(2)
22triple = make_multiplier(3)
23
24print("Double 10:", double(10))
25print("Triple 10:", triple(10))
>>>Output
5 squared: 25
5 cubed: 125
5 to the fourth: 625
Double 10: 20
Triple 10: 30
Each returned function captures its configuration value. The square function always uses exponent 2, cube uses 3. This is closure in action - the inner function remembers the outer function's variables.

Callback Pattern Example

Callbacks are functions passed to other code to be called later. This pattern is everywhere in programming:
1def process_data(data, on_success, on_error):
2 """Process data and call appropriate callback."""
3 try:
4 # Simulate processing
5 if len(data) == 0:
6 raise ValueError("Data cannot be empty")
7 result = sum(data) / len(data)
8 on_success(result)
9 except Exception as e:
10 on_error(str(e))
11
12def handle_success(result):
13 print("Success! Average is:", result)
14
15def handle_error(message):
16 print("Error occurred:", message)
17
18# Process with callbacks
19print("Processing [1,2,3,4,5]:")
20process_data([1, 2, 3, 4, 5], handle_success, handle_error)
21
22print("\nProcessing empty list:")
23process_data([], handle_success, handle_error)
24
25print("\nUsing lambdas for simple callbacks:")
26process_data([10, 20, 30],
27 lambda r: print("Got result:", r),
28 lambda e: print("Problem:", e))
>>>Output
Processing [1,2,3,4,5]:
Success! Average is: 3.0
 
Processing empty list:
Error occurred: Data cannot be empty
 
Using lambdas for simple callbacks:
Got result: 20.0
Python Quiz

> Use a higher-order function to apply a transformation to every element. Pick the built-in that applies a function to each item, and the built-in that converts the result into a usable list.

def double(x):
    return x * 2

nums = [1, 2, 3, 4]
result = ___(___(double, nums))
print(result)
sorted
tuple
list
filter
map
Treating functions as first-class values unlocks patterns that are fundamental to data engineering. Dispatch tables, callbacks, and function factories all rely on this core idea.
When you store functions in a dictionary, adding a new operation requires only a single dictionary entry. The code that calls the operation never changes, making your programs open for extension without requiring modification.

The map() and filter() built-ins are direct expressions of higher-order programming. Once you are comfortable with functions as values, these patterns become natural building blocks for clean, expressive code.

Decorators

Daily Life
Interviews

Wrap functions with @ decorators

A decorator is a function that wraps another function, adding behavior without modifying the original. Decorators combine everything you've learned: higher-order functions, closures, and function objects.
Decorators are everywhere in professional Python. Web frameworks like Flask and Django use them for routing. Testing frameworks use them for setup. Data libraries use them for caching. Understanding decorators is essential.

The Decorator Pattern

A decorator takes a function, creates a wrapper that adds behavior, and returns the wrapper. Let's build one step by step:
1def announce(func):
2 """Announces function execution."""
3 def wrapper(x):
4 print("About to run: " + func.__name__)
5 result = func(x)
6 print("Finished running: " + func.__name__)
7 return result
8 return wrapper
9
10def square(x):
11 return x * x
12
13# Manually apply the decorator
14decorated_square = announce(square)
15
16# Call the decorated version
17print("Result:", decorated_square(5))
18
19print()
20
21# Original is unchanged
22print("Original still works:", square(5))
>>>Output
About to run: square
Finished running: square
Result: 25
 
Original still works: 25

The announce decorator wraps square in a wrapper that prints messages before and after calling the original function. The wrapper is a closure that remembers func.

The @ Syntax

Python provides the @ syntax as shorthand for applying decorators. This is cleaner and makes the decoration obvious:

1def announce(func):
2 """Announces function calls."""
3 def wrapper(x):
4 print("Running:", func.__name__)
5 result = func(x)
6 return result
7 return wrapper
8
9@announce
10def double(x):
11 return x * 2
12
13@announce
14def add_ten(x):
15 return x + 10
16
17# Both are decorated
18print("Double 5:", double(5))
19print()
20print("Add 10 to 5:", add_ten(5))
>>>Output
Running: double
Double 5: 10
 
Running: add_ten
Add 10 to 5: 15

Using @announce above a function definition is exactly equivalent to writing double = announce(double) after the definition. The @ syntax is just cleaner.

Practical Decorator: Timing

One of the most useful decorators measures how long a function takes to execute:
1import time
2
3def timer(func):
4 """Decorator that times function execution."""
5 def wrapper(*args, **kwargs):
6 start = time.time()
7 result = func(*args, **kwargs)
8 end = time.time()
9 print(f"{func.__name__} took {end - start:.4f} seconds")
10 return result
11 return wrapper
12
13@timer
14def slow_function():
15 time.sleep(1)
16 return "Done"
17
18@timer
19def fast_function():
20 return sum(range(1000))
21
22slow_function()
23fast_function()
This pattern is extremely useful for performance profiling. Add the decorator to any function you want to measure, without changing the function itself.

Decorator with Validation

Decorators are perfect for validation because they separate the validation logic from the main function:
1def require_positive(func):
2 """Ensures first arg is positive."""
3 def wrapper(x):
4 if x <= 0:
5 return "Error: value must be positive"
6 return func(x)
7 return wrapper
8
9@require_positive
10def calculate_square_root(x):
11 return x ** 0.5
12
13@require_positive
14def calculate_log(x):
15 # Simplified log calculation
16 return str(x) + " is positive, log would work"
17
18print(calculate_square_root(16))
19print(calculate_square_root(25))
20print(calculate_square_root(-5))
21print()
22print(calculate_log(100))
23print(calculate_log(-10))
>>>Output
4.0
5.0
Error: value must be positive
 
100 is positive, log would work
Error: value must be positive
The validation logic lives in one place - the decorator. Any function that needs this validation just adds the decorator. This is the DRY principle (Don't Repeat Yourself) in action.

Handling Any Arguments

Real-world decorators need to work with functions that have different signatures. Use *args and **kwargs to handle any combination of arguments:

1def log_call(func):
2 def wrapper(*args, **kwargs):
3 print(f"Call: {func.__name__}({args}, {kwargs})")
4 result = func(*args, **kwargs)
5 print(f"Return: {result}")
6 return result
7 return wrapper
8
9@log_call
10def add(a, b):
11 return a + b
12
13@log_call
14def greet(name, msg="Hi"):
15 return f"{msg}, {name}!"
16
17add(3, 5)
18print()
19greet("Alice", msg="Hello")
>>>Output
Call: add((3, 5), {})
Return: 8
 
Call: greet(('Alice',), {'msg': 'Hello'})
Return: Hello, Alice!

*args collects all positional arguments as a tuple. **kwargs collects all keyword arguments as a dictionary. The wrapper passes them along to the original function unchanged.

Stacking Decorators

You can apply multiple decorators to a single function. They apply from bottom to top:
1def bold(func):
2 def wrapper(*args):
3 return "**" + func(*args) + "**"
4 return wrapper
5
6def italic(func):
7 def wrapper(*args):
8 return "_" + func(*args) + "_"
9 return wrapper
10
11def uppercase(func):
12 def wrapper(*args):
13 return func(*args).upper()
14 return wrapper
15
16@bold
17@italic
18@uppercase
19def greet(name):
20 return "hello " + name
21
22print(greet("world"))
>>>Output
**_HELLO WORLD_**
Common Decorator Uses
  • Logging: Record when functions are called
  • Timing: Measure execution duration
  • Validation: Check inputs before processing
  • Caching: Store results for repeated calls
  • Authentication: Verify user permissions
  • Retry logic: Automatically retry failed operations
One subtle but important detail makes decorators production-ready.
TIP
When writing decorators, use functools.wraps to preserve the original function's name and docstring. This helps with debugging and documentation.
A common decorator bug is forgetting to return the wrapper function. See if you can identify and fix the issue below.
Debug Challenge

> This decorator wraps a function to print a greeting, but the wrapped function's return value is lost. Fix it so the caller receives the original return value.

The decorated function prints "Hello!" but result is None instead of "I am Alice"

Decorators that swallow return values are one of the most common bugs in functional Python code. Always ensure the wrapper returns whatever the original function produces.

The @ syntax applies a decorator at definition time. Using @timer above a function is exactly equivalent to writing timer(my_function) after the definition. The shorthand keeps decoration visible and close to the function being decorated.

Decorators compose well with each other. Stacking @bold, @italic, and @uppercase above a single function applies each decorator in order from bottom to top, creating layered behavior without modifying any function's internal logic.

Recursion

Daily Life
Interviews

Solve nested problems with recursion

Recursion is when a function calls itself. This technique is elegant for problems that can be broken into smaller versions of the same problem. While it might seem strange at first, recursion is a natural fit for many algorithms.
Data engineers encounter recursion when traversing nested JSON, processing tree structures, working with file system hierarchies, or implementing divide-and-conquer algorithms. It's also a favorite topic in technical interviews.

The Two Parts of Recursion

Every recursive function must have two parts: a base case that stops the recursion, and a recursive case that calls itself with a smaller problem.
BaseStepTrust
Base
Base Case
When to stop recursing
Step
Recursive Case
Call self with smaller input
Trust
Leap of Faith
Assume recursive call works
1def countdown(n):
2 """Count to 1."""
3 # Base case
4 if n <= 0:
5 print("Done!")
6 return
7
8 print(n)
9
10 # Recurse smaller
11 countdown(n - 1)
12
13countdown(5)
>>>Output
5
4
3
2
1
Done!
Each call to countdown passes a smaller number. Eventually n reaches 0, hitting the base case and stopping. Without the base case, the function would call itself forever (until Python stops it with an error).

Return Values in Recursion

Recursive functions can compute and return values. The classic example is factorial - the product of all positive integers up to n:
1def factorial(n):
2 """Calculate n!."""
3 # Base case
4 if n <= 1:
5 return 1
6
7 # n! = n * (n-1)!
8 return n * factorial(n - 1)
9
10# Trace factorial(5):
11# 5 * factorial(4)
12# 5 * 4 * factorial(3)
13# 5 * 4 * 3 * factorial(2)
14# 5 * 4 * 3 * 2 * 1 = 120
15
16print("5! =", factorial(5))
17print("4! =", factorial(4))
18print("3! =", factorial(3))
19print("10! =", factorial(10))
>>>Output
5! = 120
4! = 24
3! = 6
10! = 3628800

factorial(5) calls factorial(4), which calls factorial(3), and so on. Each call waits for its recursive call to return before it can compute its result.

Thinking Recursively

To solve a problem recursively, ask yourself: "How can I express this in terms of a smaller version of the same problem?" Trust that the recursive call will work correctly.
1def sum_to(n):
2 """Sum of 1 + 2 + 3 + ... + n"""
3 # Base case: sum to 0 is 0
4 if n <= 0:
5 return 0
6 # Recursive: sum to n = n + sum to (n-1)
7 return n + sum_to(n - 1)
8
9def power(base, exp):
10 """Calculate base raised to exp power."""
11 # Base case: anything^0 = 1
12 if exp == 0:
13 return 1
14 # Recursive: base^exp = base * base^(exp-1)
15 return base * power(base, exp - 1)
16
17def reverse_string(s):
18 """Reverse a string recursively."""
19 # Base case: empty or single char
20 if len(s) <= 1:
21 return s
22 # Recursive: last char + reverse of rest
23 return s[-1] + reverse_string(s[:-1])
24
25print("Sum 1 to 10:", sum_to(10))
26print("2^8:", power(2, 8))
27print("Reverse 'hello':", reverse_string("hello"))
>>>Output
Sum 1 to 10: 55
2^8: 256
Reverse 'hello': olleh
Recursion Checklist
  • Identify the base case (when to stop)
  • Identify how to reduce to a smaller problem
  • Ensure each recursive call moves toward base case
  • Trust that the recursive call works correctly
  • Test with simple inputs first

Recursive Data Structures

Recursion is natural for nested data structures like lists within lists or tree-like JSON:
1def sum_nested(data):
2 """Sum all numbers in a nested list structure."""
3 total = 0
4 for item in data:
5 if isinstance(item, list):
6 # Recursive case: sum the nested list
7 total += sum_nested(item)
8 else:
9 # Base case: it's a number
10 total += item
11 return total
12
13# Nested list with multiple levels
14nested = [1, [2, 3], [4, [5, 6]], 7]
15print("Sum of nested:", sum_nested(nested))
16
17# Another example
18deep = [[[1]], [[2]], [[3]]]
19print("Sum of deep:", sum_nested(deep))
20
21def flatten(nested_list):
22 """Flatten a nested list into a single list."""
23 result = []
24 for item in nested_list:
25 if isinstance(item, list):
26 result.extend(flatten(item))
27 else:
28 result.append(item)
29 return result
30
31print("Flattened:", flatten([1, [2, [3, 4]], 5]))
>>>Output
Sum of nested: 28
Sum of deep: 6
Flattened: [1, 2, 3, 4, 5]
Processing nested JSON data in data engineering often uses exactly this pattern. The recursive function handles any depth of nesting automatically.

Recursion vs Iteration

Many problems can be solved with either recursion or loops. Each approach has trade-offs:
Recursion
  • Elegant for trees and nested structures
  • Natural for divide-and-conquer
  • Matches mathematical definitions
  • Can be memory-intensive
  • Risk of stack overflow for deep recursion
Iteration (loops)
  • More efficient for linear processing
  • Uses constant memory
  • No stack overflow risk
  • Can be harder to express some algorithms
  • Better for performance-critical code
1# Same problem: sum 1 to n
2
3def sum_recursive(n):
4 if n <= 0:
5 return 0
6 return n + sum_recursive(n - 1)
7
8def sum_iterative(n):
9 total = 0
10 for i in range(1, n + 1):
11 total += i
12 return total
13
14print("Recursive sum to 100:", sum_recursive(100))
15print("Iterative sum to 100:", sum_iterative(100))
16
17# For simple cases, iteration is usually cleaner
18# For nested structures, recursion is cleaner
>>>Output
Recursive sum to 100: 5050
Iterative sum to 100: 5050
TIP
Python has a default recursion limit of 1000 calls. For very deep recursion, you may need to increase it with sys.setrecursionlimit() or convert to iteration.

The Fibonacci Example

The Fibonacci sequence is a classic recursion example. Each number is the sum of the two before it: 0, 1, 1, 2, 3, 5, 8, 13...
1def fibonacci(n):
2 """Return the nth Fibonacci number."""
3 # Base cases: first two numbers
4 if n <= 0:
5 return 0
6 if n == 1:
7 return 1
8
9 # Recursive case: sum of two previous
10 return fibonacci(n - 1) + fibonacci(n - 2)
11
12# Print first 10 Fibonacci numbers
13print("Fibonacci sequence:")
14for i in range(10):
15 print(f"fib({i}) = {fibonacci(i)}")
>>>Output
Fibonacci sequence:
fib(0) = 0
fib(1) = 1
fib(2) = 1
fib(3) = 2
fib(4) = 3
fib(5) = 5
fib(6) = 8
fib(7) = 13
fib(8) = 21
fib(9) = 34
Python Quiz

> A recursive function computes the factorial of n. Pick the comparison that defines the base case, and the arithmetic operator that combines n with the recursive result.

def factorial(n):
    if n ___ 1:
        return 1
    return n ___ factorial(n - 1)

print(factorial(5))
*
==
>=
<=
+
Recursion becomes natural with practice. Every recursive solution follows the same pattern: identify the simplest case that needs no further work, then express the general case in terms of a smaller version of the same problem.
Recursive functions for nested data structures handle arbitrary depth automatically. A flat loop cannot traverse data of unknown nesting depth, but a recursive function works regardless of how many layers deep the structure goes.

The trade-off between recursion and iteration is always worth considering. Recursion is expressive for tree-shaped problems, while iteration is more memory-efficient for linear sequences. Python's default recursion limit of 1000 (adjustable via sys.setrecursionlimit()) is a practical guide for when to switch approaches.

Function Composition

Daily Life
Interviews

Chain functions into data pipelines

Function composition means combining simple functions to build complex operations. The output of one function becomes the input of the next. This is a core concept in functional programming.
Data pipelines are perfect examples of composition. Raw data flows through a series of transformation functions: clean, validate, transform, aggregate, format. Each function does one thing well.

Manual Composition

The simplest form of composition is calling functions in sequence, where each function takes the previous result:
1def add_prefix(text):
2 return ">> " + text
3
4def add_suffix(text):
5 return text + " <<"
6
7def capitalize(text):
8 return text.upper()
9
10# Manual composition: step by step
11message = "hello world"
12step1 = capitalize(message)
13step2 = add_prefix(step1)
14step3 = add_suffix(step2)
15print(step3)
16
17# Or chain directly (reads inside-out)
18result = add_suffix(add_prefix(capitalize("nested calls")))
19print(result)
20
>>>Output
>> HELLO WORLD <<
>> NESTED CALLS <<
Each function takes text and returns modified text. We chain them together to build the final result. The functions are reusable - you can combine them in different ways.

Creating a Compose Function

We can create a higher-order function that composes two functions into one new function:
1def compose(f, g):
2 def composed(x):
3 return f(g(x))
4 return composed
5
6def double(x):
7 return x * 2
8
9def add_one(x):
10 return x + 1
11
12def square(x):
13 return x * x
14
15double_after_add = compose(double, add_one)
16print("double(add_one(5)):", double_after_add(5))
17
18print("double then square:", compose(square, double)(3))
19print("square then double:", compose(double, square)(3))
>>>Output
double(add_one(5)): 12
double then square: 36
square then double: 18

compose(f, g) creates a new function that first applies g then applies f to the result. Note the order: g runs first (inner), f runs second (outer).

Function Composition

You can compose any number of functions by chaining compose calls or creating a more flexible version:
1def compose_all(*functions):
2 def composed(x):
3 result = x
4 for func in reversed(functions):
5 result = func(result)
6 return result
7 return composed
8
9def add_one(x):
10 return x + 1
11
12def double(x):
13 return x * 2
14
15def square(x):
16 return x * x
17
18pipeline = compose_all(square, double, add_one)
19print("Pipeline(2):", pipeline(2))
>>>Output
Pipeline(2): 36

Pipe: Left-to-Right Flow

Compose works right-to-left, which matches function nesting. But sometimes left-to-right is more intuitive. A pipe function applies functions in the order listed:
1def pipe(*functions):
2 def piped(x):
3 result = x
4 for func in functions:
5 result = func(result)
6 return result
7 return piped
8
9def add_one(x):
10 return x + 1
11
12def double(x):
13 return x * 2
14
15def to_string(x):
16 return "Result: " + str(x)
17
18process = pipe(add_one, double, to_string)
19print(process(5))
>>>Output
Result: 12

With pipe, the functions are listed in the order they execute. Many developers find this easier to read and reason about than right-to-left composition.

Real-World Data Pipeline

Composition shines in data processing. Each function handles one transformation, making the code modular and testable:
1def pipe(*functions):
2 def piped(x):
3 for func in functions:
4 x = func(x)
5 return x
6 return piped
7
8def remove_nulls(records):
9 result = []
10 for record in records:
11 if record.get("value") is not None:
12 result.append(record)
13 return result
14
15def filter_positive(records):
16 result = []
17 for record in records:
18 if record["value"] > 0:
19 result.append(record)
20 return result
21
22def calculate_average(records):
23 if not records:
24 return 0
25 total = sum(record["value"] for record in records)
26 return total / len(records)
27
28process = pipe(remove_nulls, filter_positive, calculate_average)
29
30data = [{"value": 10}, {"value": None}, {"value": -5}, {"value": 20}]
31print("Average of positive:", process(data))
>>>Output
Average of positive: 15.0
Each function in the pipeline does one thing well. You can test them individually, reuse them in other pipelines, and easily understand what each step does. This is the power of composition.

Partial Application

Partial application is a related concept: creating a new function by pre-filling some arguments. This makes composition more flexible:
1def partial(func, *fixed_args, **fixed_kwargs):
2 def wrapper(*args, **kwargs):
3 all_kwargs = {**fixed_kwargs, **kwargs}
4 return func(*fixed_args, *args, **all_kwargs)
5 return wrapper
6
7def multiply(a, b):
8 return a * b
9
10def greet(message, name, punctuation="!"):
11 return message + ", " + name + punctuation
12
13double = partial(multiply, 2)
14triple = partial(multiply, 3)
15say_hi = partial(greet, "Hi", punctuation="...")
16
17print("double(5):", double(5))
18print("triple(5):", triple(5))
19print(say_hi("Bob"))
>>>Output
double(5): 10
triple(5): 15
Hi, Bob...

Python's standard library includes functools.partial for this purpose. It is commonly used to adapt functions for composition when their signatures do not quite match.

Common Mistakes

Even experienced developers make these mistakes with advanced functional programming:
Mistakes to Avoid
  • lambda used when a named function would be clearer
  • Forgetting the base case in recursion (infinite loop)
  • Not moving toward base case with each call (infinite loop)
  • Decorators that do not return the wrapper function
  • Decorator wrappers that do not pass *args and **kwargs
  • Deep recursion exceeding Python's stack limit
  • Complex lambda expressions that should be regular functions
1# MISTAKE: Lambda that's too complex
2bad = lambda x: x if x > 0 else (x * -1 if x < 0 else 0)
3# Better as a named function:
4def absolute_value(x):
5 if x > 0:
6 return x
7 elif x < 0:
8 return x * -1
9 else:
10 return 0
11
12print("Lambda:", bad(-5))
13print("Named:", absolute_value(-5))
14
15# MISTAKE: Decorator not returning wrapper
16def broken_decorator(func):
17 def wrapper(*args):
18 print("Before")
19 result = func(*args)
20 print("After")
21 return result
22 # Forgot to return wrapper!
23 # This would make decorated functions None
24
25# Correct:
26def working_decorator(func):
27 def wrapper(*args):
28 print("Decorated!")
29 return func(*args)
30 return wrapper
31
32@working_decorator
33def greet():
34 return "Hello"
35
36print(greet())
>>>Output
Lambda: 5
Named: 5
Decorated!
Hello
01
Lambdas
Concise anonymous functions for one-time inline use
02
Decorators
Wrap functions with reusable behavior via @ syntax
03
Recursion
Functions calling themselves to solve nested problems
04
Composition
Chain simple functions into powerful pipelines
These advanced functional patterns work together in real systems. Practice making architectural decisions that combine them effectively in the scenario below.
Data Pipeline ArchitectureStep 1
>

You are building a Python data pipeline that reads JSON logs from an API, transforms each record, validates fields, and writes clean data to a database. The pipeline must handle 100,000 records daily and be easy for your team to extend with new transformations.

raw_api_logs
timestampuser_ideventpayload
2024-03-01T10:00:00u_42click{"page": "home"}
2024-03-01T10:01:00u_99purchase{"item": "widget", "amount": 29.99}
2024-03-01T10:02:00u_42click{"page": "settings"}
May 2026

You need to apply several transformations in sequence: parse timestamps, validate user IDs, extract nested payload fields, and filter bot traffic. How do you structure the transformations?

Advanced functional techniques can dramatically simplify complex data transformations. Put your skills to the test with hands-on challenges in the Python Builder.
Function composition encourages you to write small, single-purpose functions and combine them. Each function is individually testable, and the composition is easy to read because each step has a descriptive name.
Partial application complements composition by letting you pre-configure a function's arguments. Combined with pipe or compose, partial application lets you build entire data transformation pipelines from reusable building blocks.
PUTTING IT ALL TOGETHER

> You are a senior data engineer at DoorDash building a modular ETL framework where transformation steps are defined as concise inline expressions, passed as arguments to orchestration functions, automatically timed and logged by a wrapping layer, and assembled into multi-stage pipelines through composition.

lambda functions define concise single-expression transformations for inline sorting keys and field extractors without creating named functions that clutter the module namespace.
Functions as first-class objects allow each transformation step to be stored in a list, passed as an argument, and dynamically swapped at runtime without changing the orchestration code.
Decorators wrap every ETL step function with automatic timing and error-logging behavior so instrumentation is applied uniformly without modifying any transformation's core logic.
Function composition chains simple single-purpose transformations into a multi-stage pipeline so each stage receives the output of the previous one in a readable left-to-right flow.
KEY TAKEAWAYS
lambda creates anonymous functions for simple, one-time operations - use them for sorting keys and callbacks
Functions are first-class objects: store in variables, pass as arguments, return from functions
Higher-order functions accept or return other functions, enabling powerful abstractions
Decorators wrap functions using the @ syntax - use them for logging, timing, validation, and caching
Every recursive function needs a base case and must move toward it with each call
Recursion is natural for nested data structures like trees and nested lists
Function composition chains simple functions to build complex data pipelines
Use *args and **kwargs in decorator wrappers to handle any function signature

Powerful patterns with functions

Category
Python
Difficulty
advanced
Duration
42 minutes
Challenges
0 hands-on challenges

Topics covered: Lambda Functions, Functions as Objects, Decorators, Recursion, Function Composition

Lesson Sections

  1. Lambda Functions

    Lambda Syntax Lambda: Can and Cannot Do Understanding lambda limitations helps you choose when to use them. The single-expression rule is strict but makes lambdas predictable and easy to read: When to Use Lambdas The best use cases for lambdas are situations where you need a simple function for immediate, one-time use. Here's how to decide: Higher-Order Lambdas Lambdas shine brightest when used with functions that take other functions as arguments. These are called higher-order functions. You'll

  2. Functions as Objects

    In Python, functions are objects like any other value. You can store them in variables, pass them to other functions, return them from functions, and even store them in data structures. This concept is called "first-class functions." This idea might seem abstract at first, but it's fundamental to Python's flexibility. Understanding it unlocks powerful patterns like callbacks, strategies, and the decorator pattern we'll explore later. Functions Are Values A function name without parentheses refer

  3. Decorators

    A decorator is a function that wraps another function, adding behavior without modifying the original. Decorators combine everything you've learned: higher-order functions, closures, and function objects. Decorators are everywhere in professional Python. Web frameworks like Flask and Django use them for routing. Testing frameworks use them for setup. Data libraries use them for caching. Understanding decorators is essential. The Decorator Pattern A decorator takes a function, creates a wrapper t

  4. Recursion

    Recursion is when a function calls itself. This technique is elegant for problems that can be broken into smaller versions of the same problem. While it might seem strange at first, recursion is a natural fit for many algorithms. Data engineers encounter recursion when traversing nested JSON, processing tree structures, working with file system hierarchies, or implementing divide-and-conquer algorithms. It's also a favorite topic in technical interviews. The Two Parts of Recursion Every recursiv

  5. Function Composition (concepts: pyFunctools)

    Function composition means combining simple functions to build complex operations. The output of one function becomes the input of the next. This is a core concept in functional programming. Data pipelines are perfect examples of composition. Raw data flows through a series of transformation functions: clean, validate, transform, aggregate, format. Each function does one thing well. Manual Composition The simplest form of composition is calling functions in sequence, where each function takes th