Data Structures: Intermediate

Navigate and reshape nested data

Access patterns work in two directions. The expression users[0] selects a row by index, returning a dictionary. The expression ["name"] selects a column by key from that dictionary. You can chain these operations: users[0]["name"] gives "Alice" directly. This two-step access pattern is fundamental to working with tabular data in Python.

When you need to group items by a key, use a dictionary where each value is a list. This pattern is common for categorization, grouping database records by a field, and building inverted indexes. The dictionary provides O(1) lookup by group, while the list stores all members of that group. This is more efficient than scanning through a flat list every time you need items from a specific category.

The .items() method yields key-value pairs as tuples, allowing you to iterate over both the group name and its members simultaneously. This pattern enables efficient lookups: finding all engineers is O(1) dictionary access instead of O(n) scanning through every record. For large datasets, this performance difference is significant.

Direct chaining like response["data"]["user"]["profile"] works well when you know the structure exists. However, if any key in the chain is missing, Python raises a KeyError. For uncertain data structures, you need defensive access patterns.

Chaining .get() calls with empty dict defaults {} prevents KeyError exceptions. If any level is missing, the chain returns an empty dict, and subsequent .get() calls safely return their defaults. This pattern is essential when processing API responses that may have optional fields.

Single-level safety

Use d.get("key", default) to return a fallback if the key is missing

Chained navigation

Chain .get("a", {}).get("b", val) to walk nested dicts without crashing

Empty dict fallback

Use {} as the intermediate default so the next .get() has a dict to call

Final default matters

The last .get() in the chain should return your actual fallback value

This pattern of checking for key existence and initializing an empty list is so common that Python provides a cleaner way to write it. The setdefault() method combines the check and initialization into a single operation.

The .setdefault(key, default) method returns the value if the key exists, or sets it to the default and returns that default if the key is missing. This allows you to chain .append() directly, making the grouping operation a single line. This is more Pythonic than the explicit if-check pattern.

groups = {}
for item in ["a", "b", "a", "c", "b"]:
    groups.___(item, []).append(1)
print(len(groups))
total = sum(
    len(v) for v in groups.___()
)
print(total)

Build lists and dicts in one expression

A list comprehension has the form [expression for item in iterable]. It creates a new list by evaluating the expression once for each item in the iterable. The expression can be any valid Python expression: a simple variable reference, a calculation, a method call, or even a function application.

Add an if clause to filter which items are included in the result: [expression for item in iterable if condition]. Only items where the condition evaluates to True are processed and included. The condition is evaluated before the expression, so you can safely access properties that might not exist on filtered-out items.

users = [
    {{"name": "Alice", "active": True, "role": "engineer"}},
    {{"name": "Bob", "active": False, "role": "analyst"}},
    {{"name": "Charlie", "active": True, "role": "engineer"}},
]
result = [___u["name"]u for u in users if ___u["active"] and u["role"] == "engineer"u["active"]]
print(result)

Dictionary comprehensions use curly braces with a key-value pair: {key_expr: value_expr for item in iterable}. They are essential for transforming dictionaries, filtering dictionary entries, building lookups from lists, and inverting key-value relationships.

The zip() function pairs elements from two iterables, creating tuples that you can unpack in the comprehension. Combined with a dict comprehension, it creates dictionaries from parallel lists in one line. This is a very common pattern for data transformation.

Inversion only works cleanly when values are unique. If multiple keys share the same value, later entries overwrite earlier ones. When values are not unique, you would need to group keys into lists using the setdefault pattern or defaultdict.

Set comprehensions use curly braces like dictionaries, but with single values instead of key-value pairs: {expression for item in iterable}. They automatically deduplicate results, making them perfect for extracting unique values from collections.

Compare datasets with set math

Sets support mathematical operations that are invaluable for data comparison and analysis tasks. Union combines elements from multiple sets. Intersection finds elements common to all sets. Difference finds elements unique to one set. Symmetric difference finds elements in either set but not both. These operations execute in near-constant O(n) time regardless of set size, making them dramatically more efficient than nested loops for comparison tasks.

Union returns all unique elements from both sets combined. Duplicates are automatically removed since sets only store unique values. Use the | operator or the .union() method. The method form accepts any iterable, not just sets.

Intersection returns only elements present in all sets. Use the & operator or the .intersection() method. This operation is symmetric: A & B equals B & A.

Intersection is invaluable for access control analysis, skill matching, finding common customers between segments, and identifying overlapping records between datasets. The O(1) lookup time of sets makes these operations efficient even for large datasets.

Difference returns elements in the first set that are not in the second. Use the - operator or the .difference() method. Unlike union and intersection, difference is not symmetric: A - B is different from B - A.

Symmetric difference returns elements in either set but not both, using the ^ operator. Subset and superset checking use <= and >= operators to test containment relationships.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

source = {1, 2, 3, 4, 5}
warehouse = {2, 4, 6}
missing = warehouse - source
print("Missing:", missing)

Sort and filter records like SQL

The general pattern for data queries is: filter to select relevant records, sort to order them appropriately, then optionally slice to limit results. This mirrors the SELECT...WHERE...ORDER BY...LIMIT pattern in SQL and appears constantly in data processing code.

The sorted() function accepts a key parameter that specifies how to extract a comparison value from each element. The key function is called once per element, and elements are sorted based on the returned values. This enables sorting complex objects by any attribute or computed property.

The key function is called once per element to extract the sort value. The reverse=True parameter sorts in descending order. Lambda functions are commonly used for simple key extraction, but you can use any callable.

To sort by multiple criteria, return a tuple from the key function. Python compares tuples element by element, creating a natural multi-level sort. The first element is the primary sort key, the second is the secondary sort key used to break ties, and so on.

The tuple (r["dept"], -r["years"], r["name"]) sorts first by department alphabetically, then by years in descending order (negation inverts numeric sorting), then by name to break any remaining ties. This gives you fine-grained control over sort order.

words = ["banana", "fig", "apple", "kiwi"]
result = ___(words, key=___)
print(result[0])
print(result[-1])

The itertools.groupby() function groups consecutive elements that share the same key value. Important: the data must be sorted by the grouping key first, because groupby only groups consecutive matches. This enables SQL-like GROUP BY operations on sorted data.

The groupby() function yields (key, group_iterator) pairs. The group is an iterator, not a list, so you must convert it with list(group) if you need to iterate it multiple times or access its length. This pattern enables powerful aggregations similar to SQL GROUP BY.

Choose structures by access pattern

When checking if an item exists in a collection, sets are dramatically faster than lists. Lists scan sequentially from the beginning, making membership testing O(n). Sets use hash tables for near-instant O(1) lookups. For small collections the difference is negligible, but for thousands of items, sets can be hundreds of times faster.

When you need to find records by a specific field, dictionaries provide O(1) lookup while lists require O(n) scanning. If you frequently look up records by ID, name, or any other unique key, building a dictionary indexed by that key transforms slow linear searches into instant hash lookups.

Building the dictionary is O(n), but each subsequent lookup is O(1). If you look up records more than once, the preprocessing cost is repaid. For APIs or data processing that repeatedly access records by key, always build lookup dictionaries.

Tuples are immutable sequences. Use them when you need a fixed record that should not be modified, like coordinates, RGB colors, or database rows. Tuples use slightly less memory than lists and, critically, can be used as dictionary keys since they are hashable.

Named tuples from collections.namedtuple provide the benefits of tuples (immutability, hashability) with the readability of named fields. For modern Python, consider @dataclass(frozen=True) which offers similar benefits with additional features like default values and type hints.

Sets use significantly more memory due to their hash table structure, but this overhead enables O(1) membership testing. The trade-off is worthwhile when lookup speed matters more than memory. Tuples are slightly smaller than lists because they do not allocate extra space for potential growth.

ids = [101, 202, 303, 404]
lookup = ___(ids)
print(202 ___ lookup)
print(len(lookup))