Collections: Intermediate

Sort any data by custom criteria

The sorted() function returns a new sorted list from any iterable. While basic sorting works on simple values, the real power comes from the key parameter. The key function transforms each element into a comparison value, letting you sort by any criteria you can express in code.

In SQL, you use ORDER BY to sort query results. The key parameter in sorted() serves the same purpose: it defines the sorting expression. When you write sorted(users, key=lambda u: u["age"]), you are essentially writing ORDER BY age in Python. This conceptual connection helps when you move between SQL queries and Python data processing.

The sorted() function always returns a new list, leaving the original unchanged:

The key parameter accepts a function that transforms each element into a comparison value. Python sorts based on these transformed values:

The lambda creates an anonymous function inline. The expression lambda u: u["age"] means "given a user u, return their age." This extracted value is what Python uses for comparison.

This tuple-based sorting mirrors ORDER BY with multiple columns in SQL. When you write sorted(employees, key=lambda e: (e[1], e[2])), you are expressing ORDER BY department, salary. The first element of the tuple is the primary sort key; subsequent elements only matter when earlier elements are equal. You can have as many tiebreaker levels as needed.

Performance matters when sorting large datasets. Python's Timsort algorithm is highly optimized, running in O(n log n) time. The key function is called exactly once per element (O(n) total), and the results are cached. This means even complex key computations do not dramatically slow down sorting. However, if your key function itself does O(n) work, the overall complexity becomes O(n²), so keep key functions simple.

len

Sort strings or collections by their length

str.lower

Case-insensitive sorting by converting to lowercase for comparison

abs

Sort numbers by absolute value, ignoring sign

lambda x: x[n]

Sort tuples by a specific positional element

lambda d: d["field"]

Sort dictionaries by a named field value

lambda x: (x.a, x.b)

Multi-level sort with tuple keys for tiebreaking

words = ["banana", "fig", "cherry", "kiwi"]
result = sorted(words, key=___lenstr.lowerstr.upper)
print(result)

Rank items and find top N results

The reverse parameter controls whether sorted() returns results in ascending (default) or descending order. Setting reverse=True flips the sort order completely without requiring you to reverse the resulting list afterward.

The key and reverse parameters work together seamlessly. The key determines what to compare; reverse determines the order. You can use both simultaneously for powerful sorting combinations:

Timsort is particularly efficient when data has "runs" of already-sorted elements. Real-world data often has this structure, whether from timestamps, alphabetical lists, or numeric sequences. The algorithm detects these runs and merges them efficiently. This is why Python's sorted() often outperforms theoretical O(n log n) expectations on practical data. The same algorithm was adopted by Java 7 and the V8 JavaScript engine for their standard sorting implementations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

scores = [88, 72, 95, 61, 84]
top_3 = scores.sort(reverse=True)[:3]
print("Top 3:", top_3)

The key distinction between sorted() and .sort() is their return value. sorted() always returns a new list; .sort() modifies in place and returns None. Assigning .sort() to a variable is a common bug.

When you need the top or bottom N elements from a large collection, combining sorted() with slicing is simple and readable. For very large N, heapq.nlargest() or heapq.nsmallest() offers better performance.

The reverse parameter works together with key seamlessly. Python applies the key first to determine comparison values, then reverses the order of those results to produce descending output.

Transform every element without a loop

The map() function applies a transformation function to every element in an iterable, returning a map object that yields the results lazily. This is one of Python's core functional programming tools, enabling concise data transformations without explicit loops.

In data engineering, map() is used constantly. Converting a column of strings to integers, extracting fields from records, formatting values for output, all of these are map operations. Understanding map() leads to cleaner code and prepares you for distributed processing frameworks like PySpark where map operations are fundamental.

Map operations have a mathematical foundation in category theory, but you do not need to understand the theory to use them effectively. What matters is recognizing the pattern: when you want to apply the same operation to every element of a collection, map() is usually cleaner than a loop. The pattern is so universal that SQL (SELECT expression), spreadsheets (formulas applied to columns), and big data tools (Spark's map) all have equivalent concepts.

Notice that map() returns a map object, not a list. We wrap it in list() to see all the results. This lazy evaluation means map() is memory-efficient for large datasets since it processes one element at a time.

The map() function can take multiple iterables. The function receives one element from each iterable:

prices = ["19.99", "29.99", "9.99"]
amounts = list(map(___, prices))
total = ___(amounts)
print(round(total, 2))

map() is most readable when the transformation is a named function. Passing int, float, or str.upper directly is cleaner than wrapping them in a lambda.

map() returns a lazy iterator, meaning it only computes each result when requested. This makes it memory-efficient for large collections since the entire transformed dataset is never held in memory at once.

Select matching records from any list

The filter() function selects elements from an iterable where a predicate function returns True. Like map(), it returns an iterator that yields results lazily. This is Python's built-in way to extract subsets of data matching specific criteria.

Filtering is fundamental to data processing. Every query that includes a WHERE clause is a filter operation. Every data validation step that removes invalid records is filtering. Understanding filter() helps you think declaratively about data selection and prepares you for SQL and data pipeline tools that use the same concepts.

The predicate function you pass to filter() is a decision maker. For each element, it answers yes or no: should this element be included in the result? This binary decision mirrors the boolean logic of SQL WHERE clauses. In fact, when you write filter(lambda x: x > 5, numbers), you are expressing WHERE x > 5 in Python. The conceptual model is identical across languages and tools.

Passing None as the function removes falsy values (empty strings, zero, None, empty lists, etc.):

Complex predicates can combine multiple conditions with and/or. This mirrors WHERE clauses in SQL. The lambda expresses the condition declaratively, making the intent clear without loop mechanics.

nums = [-2, 3, 4, -5, 6, 7, 8]
result = list(filter(___lambda x: x > 0 and x % 2 == 0lambda x: x % 2 == 0lambda x: x > 0, nums))
print(result)

Merge parallel lists into paired records

The zip() function combines multiple iterables element-by-element, creating tuples of corresponding items. It's like a zipper that interleaves teeth from both sides. This is essential for working with parallel data structures where related information is stored in separate sequences.

Without zip(), you would need to iterate using indices: for i in range(len(names)). This approach is error-prone because you might index out of bounds if lists have different lengths, and it is harder to read. The zip() version directly expresses the intent: iterate over these collections together, giving me one element from each at a time.

The unpacking in the for loop, for name, age, city in zip(...), is called tuple unpacking. Each iteration, zip() yields a tuple containing one element from each input. The for statement unpacks that tuple directly into named variables. This combination of zip() and tuple unpacking is idiomatic Python that you will see in professional codebases everywhere.

By default, zip() stops when the shortest iterable is exhausted. Extra elements in longer iterables are silently ignored. This behavior is often surprising to beginners and can cause subtle bugs if you do not expect it:

The itertools.zip_longest() function continues until the longest iterable is exhausted, filling missing values with the specified fillvalue. This is crucial when you need to preserve all data even when sources have different lengths.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

keys = ["name", "age", "city"]
values = ["Alice", 30, "Seattle"]
record = zip(keys, values)
print(record["name"])

You can "unzip" a list of tuples back into separate lists using zip(*pairs). The * unpacks the list, passing each tuple as a separate argument. This reverses the zip operation, separating combined data back into its component sequences:

Remember: sorted() is a function that returns a new list; .sort() is a method that modifies in place and returns None. A common bug is writing x = mylist.sort() which makes x equal to None.

keys = ["name", "age", "role"]
vals = ["Alice", 30, "Engineer"]
record = ___(___(keys, vals))
print(record["role"])

zip() is a fundamental tool for working with parallel data. Whenever you have related information stored in separate sequences, zip() lets you iterate over them together as if they were rows in a table.