Helper Decomposition

Concepts covered: Functions

When one function body parses and validates each row, then converts it and reports the result, the only way to check it is to build a whole batch to run through it and then read the printout. When the total comes out wrong, the fault could sit anywhere in it. Here is such a function for payment rows, with each rate stored as US cents per 100 units of the row's currency.

def process_batch(rows, rates):
    total = 0
    rejected = 0
    for row in rows:
        try:
            cents = int(row["amount"])
        except ValueError:
            rejected += 1
            continue
        if cents < 0:
            rejected += 1
            continue
        if row["currency"] not in rates:
            rejected += 1
            continue
        total += cents * rates[row["currency"]] // 100
    print("total usd cents:", total)
    print("rejected:", rejected)

Cut along the jobs

Name each job in the body with a short phrase. Here the first job turns a row into a non-negative number of cents, and the second checks that a rate exists before converting. A third counts good and bad rows to hand back the result. Each phrase becomes a helper that takes what it needs as parameters and either returns its result or raises when it cannot produce one. The original function shrinks to a coordinator that calls the helpers and decides what a failure means for the batch.

def read_amount(row):
    cents = int(row["amount"])
    if cents < 0:
        raise ValueError("negative amount")
    return cents

def to_usd_cents(cents, currency, rates):
    if currency not in rates:
        raise ValueError("no rate for currency")
    return cents * rates[currency] // 100

def process_batch(rows, rates):
    total = 0
    rejected = 0
    for row in rows:
        try:
            cents = read_amount(row)
            total += to_usd_cents(cents, row["currency"], rates)
        except ValueError:
            rejected += 1
    return total, rejected

rates = {"usd": 100, "eur": 108}
rows = [
    {"amount": "1200", "currency": "eur"},
    {"amount": "-350", "currency": "usd"},
    {"amount": "500", "currency": "gbp"},
    {"amount": "800", "currency": "usd"},
]
print(process_batch(rows, rates))
print(to_usd_cents(1200, "eur", rates) == 1296)
print(read_amount({"amount": "0", "currency": "usd"}) == 0)
try:
    to_usd_cents(500, "gbp", rates)
except ValueError as err:
    print("raised:", err)

Output:

(2096, 2)
True
True
raised: no rate for currency

The coordinator has a single reject path where it had 3: any helper that raises ValueError sends the row to the same except. It returns (2096, 2), a pair other code can use. The 1296 inside that total is 1200 euro cents at 1.08 dollars per euro, rounded down by //.

Each row passes through 2 helpers in turn, and a ValueError from either becomes a reject in the coordinator's count
  • payment rows (source, list of dicts)
  • read_amount (transform, helper); on error: raise ValueError
  • to_usd_cents (transform, helper); on error: raise ValueError
  • process_batch (consumer, coordinator); on error: count as rejected
  • payment rows -> read_amount
  • read_amount -> to_usd_cents
  • to_usd_cents -> process_batch

Each helper gets its own check

The last 3 checks in that program each test a single rule with values typed in place, including the failure path for a currency with no rate. Before the split, testing the conversion meant building a row inside a batch and reading printed text. Now a broken rule points at the helper that owns it.

The 18-line version
  • Checking the conversion means building a batch and reading printed text
  • Separate reject paths in 3 places, each with its own continue
  • Prints the totals, so no other code can use them
  • A new rule means editing the middle of the loop
Helpers plus a coordinator
  • Each rule is checked by a line with values typed in place
  • A single reject path: any ValueError from any helper
  • Returns (total, rejected) for other code to use
  • A new rule is a new raise in the helper that owns it

Where helpers go wrong

Splitting can go too far, or cut in the wrong place. A helper that needs 6 parameters is usually 2 jobs, or a piece separated from the data it works on. Reading a global variable means the helper can't be checked by its arguments alone, because the same call returns different values as the global changes. Printing the result instead of returning it leaves a check by eye as the only option. A good cut is a helper you can describe in a short phrase with no "and" in it, and check with 1 line.

The program prints 450 and then a second line. What's the second line, and why?

discount_pct = 10

def final_price(cents):
    price = cents - cents * discount_pct // 100
    print(price)

print(final_price(500) == 450)

About this lesson

This section is part of the Functions: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.