Raising Errors From a Function

Concepts covered: Functional Transforms

A function makes a promise through its return value: parse_cents promises a whole number of cents. When a row holds "n/a", the function can't keep that promise, and it has 2 choices. It can return a stand-in such as -1 or None and hope every caller checks for it, or it can raise an exception and return nothing at all.

What a quiet failure costs

def parse_cents_quiet(raw):
    try:
        return int(raw)
    except ValueError:
        return -1

total = 0
for raw in ["1200", "350", "n/a", "800"]:
    total += parse_cents_quiet(raw)
print("total cents:", total)

Output:

total cents: 2349

The readable rows add up to 2350 cents, but the -1 blended into the sum: the report is off by 1 cent, the "n/a" row vanished, and no line of output says anything went wrong. Returning None fails in a different place. The += line raises TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType', which points at the loop instead of the bad row, and every caller has to remember an is None check to avoid it.

A raise stops the function and travels up

raise ValueError("message") creates an exception object and throws it. The function stops on that line and returns nothing, and the exception goes to whoever called it. If that caller wrapped the call in a matching try/except, the except branch runs. If not, the exception climbs to the caller's caller, and on up, until something catches it or the program ends with a traceback.

def parse_cents(raw):
    cents = int(raw)
    if cents < 0:
        raise ValueError("amount cannot be negative")
    return cents

for raw in ["1200", "-350", "n/a", "800"]:
    try:
        print(raw, "->", parse_cents(raw))
    except ValueError as err:
        print(raw, "-> rejected:", err)

Output:

1200 -> 1200
-350 -> rejected: amount cannot be negative
n/a -> rejected: invalid literal for int() with base 10: 'n/a'
800 -> 800

int("n/a") raises its own ValueError, and parse_cents lets it pass through because it has nothing to add. A negative amount converts without complaint, so the rule against it has to come from the function itself, which raises. The loop decides what a bad row means for this job; here it reports the row and moves on.

def parse_cents(raw):
    cents = int(raw)
    if cents < 0:
        raise ValueError("amount cannot be negative")
    return cents

def load_batch(rows):
    good = 0
    rejected = 0
    for raw in rows:
        try:
            good += parse_cents(raw)
        except ValueError:
            rejected += 1
    return good, rejected

print(load_batch(["1200", "-350", "n/a", "800"]))
(2000, 2)

load_batch returns (2000, 2): 2000 cents from the 2 clean rows and a reject count the job can log or alert on. A wrong number never passes for a right one anywhere in the run.

What does this program print?

def check(n):
    if n > 3:
        raise ValueError("too big")
    print("ok", n)
    return n

total = 0
for n in [1, 5, 2]:
    try:
        total += check(n)
    except ValueError:
        print("caught")
print(total)

Choosing the exception type

SituationRaiseExample message
Right type, unacceptable value`ValueError`amount cannot be negative
Argument of the wrong type`TypeError`rows must be a list
Lookup key is missing`KeyError`no rate for gbp
A rule specific to your dataa subclass you definerefund exceeds original charge

Callers catch by type, so the type is part of the function's interface. A custom exception is a class that names an existing exception in parentheses, which makes it a kind of that exception. After class RefundTooLarge(ValueError): pass, an except ValueError still catches it, and a caller who cares about this case alone can write except RefundTooLarge. The message names the rule that broke, for whoever reads the log.

class RefundTooLarge(ValueError):
    pass

def refund_cents(charged, requested):
    if requested > charged:
        raise RefundTooLarge("refund exceeds original charge")
    return requested

for requested in [500, 3000]:
    try:
        print("refunded", refund_cents(2500, requested))
    except RefundTooLarge as err:
        print("blocked:", err)

Output:

refunded 500
blocked: refund exceeds original charge
Do
  • Raise as soon as the function knows it can't return a correct value
  • Pick the type a caller would catch: ValueError, TypeError, KeyError or a subclass
  • Write a message that names the rule the input broke
  • Catch where someone can decide between skipping or quarantining the row and stopping the job
Avoid
  • Signal failure by returning -1 or 0, or by returning an empty string
  • Catch an error inside a helper only to return None
  • Wrap a call in except Exception and carry on without a record
  • Use the same exception type for every failure in a module

About this lesson

This section is part of the Functions: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.