Helper Decomposition
Concepts covered: Functions
When one function body parses and validates each row, then converts it and reports the result, the only way to check it is to build a whole batch to run through it and then read the printout. When the total comes out wrong, the fault could sit anywhere in it. Here is such a function for payment rows, with each rate stored as US cents per 100 units of the row's currency.
def process_batch(rows, rates):
total = 0
rejected = 0
for row in rows:
try:
cents = int(row["amount"])
except ValueError:
rejected += 1
continue
if cents < 0:
rejected += 1
continue
if row["currency"] not in rates:
rejected += 1
continue
total += cents * rates[row["currency"]] // 100
print("total usd cents:", total)
print("rejected:", rejected)Cut along the jobs
Name each job in the body with a short phrase. Here the first job turns a row into a non-negative number of cents, and the second checks that a rate exists before converting. A third counts good and bad rows to hand back the result. Each phrase becomes a helper that takes what it needs as parameters and either returns its result or raises when it cannot produce one. The original function shrinks to a coordinator that calls the helpers and decides what a failure means for the batch.
def read_amount(row):
cents = int(row["amount"])
if cents < 0:
raise ValueError("negative amount")
return cents
def to_usd_cents(cents, currency, rates):
if currency not in rates:
raise ValueError("no rate for currency")
return cents * rates[currency] // 100
def process_batch(rows, rates):
total = 0
rejected = 0
for row in rows:
try:
cents = read_amount(row)
total += to_usd_cents(cents, row["currency"], rates)
except ValueError:
rejected += 1
return total, rejected
rates = {"usd": 100, "eur": 108}
rows = [
{"amount": "1200", "currency": "eur"},
{"amount": "-350", "currency": "usd"},
{"amount": "500", "currency": "gbp"},
{"amount": "800", "currency": "usd"},
]
print(process_batch(rows, rates))
print(to_usd_cents(1200, "eur", rates) == 1296)
print(read_amount({"amount": "0", "currency": "usd"}) == 0)
try:
to_usd_cents(500, "gbp", rates)
except ValueError as err:
print("raised:", err)Output:
(2096, 2) True True raised: no rate for currency
The coordinator has a single reject path where it had 3: any helper that raises ValueError sends the row to the same except. It returns (2096, 2), a pair other code can use. The 1296 inside that total is 1200 euro cents at 1.08 dollars per euro, rounded down by //.
- payment rows (source, list of dicts)
- read_amount (transform, helper); on error: raise ValueError
- to_usd_cents (transform, helper); on error: raise ValueError
- process_batch (consumer, coordinator); on error: count as rejected
- payment rows -> read_amount
- read_amount -> to_usd_cents
- to_usd_cents -> process_batch
Each helper gets its own check
The last 3 checks in that program each test a single rule with values typed in place, including the failure path for a currency with no rate. Before the split, testing the conversion meant building a row inside a batch and reading printed text. Now a broken rule points at the helper that owns it.
- The 18-line version
- Checking the conversion means building a batch and reading printed text
- Separate reject paths in 3 places, each with its own continue
- Prints the totals, so no other code can use them
- A new rule means editing the middle of the loop
- Helpers plus a coordinator
- Each rule is checked by a line with values typed in place
- A single reject path: any ValueError from any helper
- Returns (total, rejected) for other code to use
- A new rule is a new raise in the helper that owns it
Where helpers go wrong
Splitting can go too far, or cut in the wrong place. A helper that needs 6 parameters is usually 2 jobs, or a piece separated from the data it works on. Reading a global variable means the helper can't be checked by its arguments alone, because the same call returns different values as the global changes. Printing the result instead of returning it leaves a check by eye as the only option. A good cut is a helper you can describe in a short phrase with no "and" in it, and check with 1 line.
The program prints 450 and then a second line. What's the second line, and why?
discount_pct = 10
def final_price(cents):
price = cents - cents * discount_pct // 100
print(price)
print(final_price(500) == 450)- False: final_price prints 450 but returns None, and None == 450 is false
- True: the function computes 450, and that value is compared with 450
- False: the global discount_pct cannot be read inside the function
- A TypeError: == cannot compare a number with None
About this lesson
This section is part of the Functions: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.