Using map() for Transforms

Concepts covered: Functional Transforms

map(function, iterable) calls the function on each item and produces the results in the same order. Raw extracts often deliver numbers as text, and map(int, raw) converts a whole column in a single expression. Predict both printed lines before you look:

raw = ["42", "7", "19"]
minutes = map(int, raw)
print(list(minutes))
print(list(minutes))

Output:

[42, 7, 19]
[]

map() returns a map object, which produces each value only when something asks for it, and produces it once. list() asks for every value, so the second list() call finds nothing left and returns []. Store the result of list() in a variable whenever you need the values twice.

A job converts row counts, totals them, then checks how many it got. What does this print?

raw = ["1200", "0", "98000"]
counts = map(int, raw)
total = sum(counts)
print(total, len(list(counts)))

The function runs when the values are read

Creating the map object converts nothing. Step through this program and watch when each converting line prints relative to the loop.

def to_minutes(text):
    print("converting", text)
    return int(text)


raw = ["42", "7", "19"]
total = 0
print("loop starts, nothing converted yet")
for value in map(to_minutes, raw):
    total += value
    print("running total", total)
print("done:", total)
loop starts, nothing converted yet
converting 42
running total 42
converting 7
running total 49
converting 19
running total 68
done: 68

If map() converted eagerly, all 3 converting lines would print before the first running total. Instead each text value is converted only when the loop asks for it, with the running total updated in between. That timing also decides where an error surfaces: a bad value fails on the line that reads the map, which can be far from the line that built it.

raw = ["42", "n/a", "19"]
minutes = map(int, raw)
print("map created")
print(list(minutes))

Run it and map created prints first, because building the map converted nothing. The failure comes on line 4, when list() reads the values and int('n/a') runs: the traceback names line 4 and ends with ValueError: invalid literal for int() with base 10: 'n/a'. Line 2, where map() was written, never appears in it. When a step fails inside a list() or sum() call, trace the map object back to the data it was built from.

Transforming records with your own function

Any function that takes 1 item and returns 1 value works with map(). For records, the function usually builds a new dict from the old one, so the source list stays intact for any other step that reads it.

def clean(user):
    return {"id": user["id"], "email": user["email"].lower()}


users = [
    {"id": 1, "email": "Ana@Site.com"},
    {"id": 2, "email": "BO@site.COM"},
]
for user in map(clean, users):
    print(user)

Output:

{'id': 1, 'email': 'ana@site.com'}
{'id': 2, 'email': 'bo@site.com'}

A map object is iterable like a list, so one map can feed another. str.strip and str.upper are methods named through their type, and they plug in the same way str.lower did as a sort key. Here country codes arrive padded and in mixed case:

raw_codes = [" pt", "BR ", "us"]
codes = list(map(str.upper, map(str.strip, raw_codes)))
print(codes)

Output:

['PT', 'BR', 'US']

Neither map does anything until list() reads the outer one. Then each code passes through both steps before the next code starts: ' pt' becomes 'pt', then 'PT'. Keep the list that list() returns, because both map objects are spent after that one read.

Given 2 iterables, map() pairs them up by position and calls the function with 1 item from each. It stops when the shorter one runs out:

def line_total(price, quantity):
    return price * quantity


prices = [5, 12, 3]
quantities = [2, 1, 4]
print(list(map(line_total, prices, quantities)))

Output:

[10, 12, 12]
A for loop with append
  • Needs an empty list set up before the loop that appends to it
  • Has room for several statements and if branches
  • Builds the full list right away
  • Reads well when the transform is written inline
map()
  • A single expression around a named function
  • The function takes 1 item per iterable and returns 1 value
  • Produces values only when read, and only once
  • Reads well when you pass a built-in like int or a method like str.lower

About this lesson

This section is part of the Collections: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.