Using sorted() with key

Concepts covered: Sorting

sorted() takes any iterable, compares its items with <, and returns a new list in ascending order. The input is left as it was, so other code that still reads it sees the original order.

durations = [42, 7, 19, 7, 63]
ordered = sorted(durations)
print(ordered)
print(durations)

Output:

[7, 7, 19, 42, 63]
[42, 7, 19, 7, 63]

Strings compare character by character using each character's code, and the uppercase letters A to Z all have lower codes than the lowercase letters a to z. A mixed-case list of table names therefore comes back in an order nobody would pick by hand. The key argument fixes that:

tables = ["orders", "Users", "events", "Accounts"]
print(sorted(tables))
print(sorted(tables, key=str.lower))

Output:

['Accounts', 'Users', 'events', 'orders']
['Accounts', 'events', 'orders', 'Users']

What the key function does

key takes a function. sorted() calls it once on each item, sorts by the values it returns, and hands back the original items in that order. With key=str.lower, the comparisons run on 'users' and 'accounts', yet the output still reads 'Users' and 'Accounts'. The key decides the order and never changes what is in the list.

Any function that takes 1 argument can be a key, built-ins included. With key=len, table names line up by character count. events and orders both have 6 characters, and events stays ahead because it came first in the input:

tables = ["events", "orders", "Users", "Accounts", "dim"]
print(sorted(tables, key=len))

Output:

['dim', 'Users', 'events', 'orders', 'Accounts']

Pipeline records are usually dicts, and 2 dicts cannot be compared with <, so you write a small function that returns the field to sort on. This program prints a line each time its key function runs. Compare the order of those calls with the order of the final output.

def by_minutes(run):
    print("key for", run["job"])
    return run["minutes"]


runs = [
    {"job": "load_orders", "minutes": 42},
    {"job": "dedupe_users", "minutes": 7},
    {"job": "build_daily", "minutes": 19},
]
ordered = sorted(runs, key=by_minutes)
for run in ordered:
    print(run["job"], run["minutes"])

key for load_orders
key for dedupe_users
key for build_daily
dedupe_users 7
build_daily 19
load_orders 42

All 3 key calls happen first, in input order, before any item moves; the runs then come out by their returned minutes. The argument is key=by_minutes, with no parentheses, which hands sorted() the function itself. Writing key=by_minutes() would call it on the spot with no run to read. The same slip with a built-in, key=len(), fails before any sorting with TypeError: len() takes exactly one argument (0 given).

Sorting on 2 fields with a tuple key

Tuples compare the way lists do: first items first, and the second items only when the first ones tie. A key function that returns a tuple therefore sorts by the first field and breaks ties with the next. Here the runs group by team, and within a team the shortest run leads.

runs = [
    {"job": "load_orders", "team": "sales", "minutes": 42},
    {"job": "dedupe_users", "team": "growth", "minutes": 7},
    {"job": "build_daily", "team": "sales", "minutes": 19},
    {"job": "sync_crm", "team": "growth", "minutes": 7},
]


def team_then_minutes(run):
    return (run["team"], run["minutes"])


for run in sorted(runs, key=team_then_minutes):
    print(run["team"], run["minutes"], run["job"])

Output:

growth 7 dedupe_users
growth 7 sync_crm
sales 19 build_daily
sales 42 load_orders

Both growth runs have the key ('growth', 7). With the keys fully tied, dedupe_users stays ahead of sync_crm because it came first in the input. Python's sort is stable: items with equal keys keep their original relative order. That guarantee is what makes a sorted report come out the same on every run over the same data.

Missing values break the comparison

Real columns have gaps, and Python has no rule for comparing None with a number. Decide what this program prints before you check the answer.

A latency column has a missing reading. What does this program print?

latencies = [120, None, 85]
print(sorted(latencies))

• It prints [None, 85, 120], with the missing value placed first
• It prints [85, 120, None], with the missing value placed last
• It raises a TypeError, because None and int cannot be compared
• It prints [85, 120], with the missing value dropped from the list

The fix is a key that never puts None and a number into the same comparison. Return a tuple whose first item marks the value as present (0) or missing (1). Every missing value then sorts after every real one, and the second items are only compared between values of the same kind.

def nulls_last(value):
    if value is None:
        return (1, 0)
    return (0, value)


latencies = [120, None, 85]
print(sorted(latencies, key=nulls_last))

Output:

[85, 120, None]

About this lesson

This section is part of the Collections: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.