Dictionaries: Intermediate

A for loop over a dict yields its keys. 3 methods choose something else for each pass: .keys() yields keys, .values() yields values, and .items() yields (key, value) tuples that the loop header unpacks into 2 names. All of them walk entries in insertion order, guaranteed since Python 3.7, meaning the order in which each key was first added. For sorted order, call sorted() explicitly.

Choose by what the loop body reads. Summing amounts needs .values(). Comparing column names needs the bare loop or .keys(), which yield the same keys in the same order, so teams pick one by convention. Anything that uses both halves of an entry needs .items(). Looping over keys and reading row[key] inside also works, but it pays a second hash lookup per entry and hides from the next reader that the body wanted pairs.

The 3 methods return view objects (dict_keys, dict_values, dict_items) that hold no copy of the data. A view reads the dict each time it is used, so creating one costs the same for 4 keys as for 4 million, and it shows keys added after the view was made.

list(statuses) copied the keys at that moment, so the later pending key never reached snapshot. A snapshot is what code needs before indexing (statuses[0] raises TypeError: 'dict_keys' object is not subscriptable) and before adding or removing keys while looping. Changing a dict's size during a loop over its live view raises RuntimeError, a trap the Advanced tier covers in its gotchas section.

A keys view tests membership with in at hash speed and supports the set operators &, |, - and ^, each returning a real set. That turns a schema drift check into 2 subtractions. A values view supports neither: 42.5 in row.values() compares against every value in turn, so its cost grows with the dict.

d.update(other) writes every pair from other into d. A key already present keeps its position and takes the new value; a new key goes on the end. The argument can be a dict, any iterable of 2-item pairs, or keyword arguments. The method changes d in place and returns None, so row = row.update(...) throws the dict away and leaves row bound to None.

d.setdefault(key, default) returns d[key] when the key exists. When it does not, it stores default under that key first and returns the stored object. .get(key, default) returns the same value in both cases but never writes, and that write is the whole difference: after setdefault, the key is in the dict.

Keeping the first value per key is one use. The other, returning a mutable object that the caller then fills, drives the grouping pattern in the nested-data section. The default argument is evaluated on every call, even when the key already exists, so setdefault(key, load_lookup()) calls load_lookup() once per record whether it is needed.

Read the chain left to right as lowest priority to highest: the layer that must win goes last. Key position follows a separate rule. A key keeps the slot of its first appearance and only its value changes, so {"a": 1, "b": 2} | {"b": 9, "a": 7} gives {'a': 7, 'b': 9}, in the left dict's order. Code that writes merged dicts out as CSV rows gets its column order from the defaults.

A module-level DEFAULTS lives as long as the process. A runner that loops over 3 jobs and calls DEFAULTS.update(job_overrides) leaves job 1's retries: 0 in place when job 2 builds its config, and job 2 never asked for it. Tests that run one job per process never catch this. | and {**a, **b} build a fresh dict on every call, so the shared base never changes. update and |= belong on dicts the current code owns, such as one it built a line earlier.

{**a, **b} also accepts literal keys after the unpacks: {**DEFAULTS, **env, "run_id": run_id} stamps a value no earlier layer can override. | requires a dict on both sides and raises TypeError for a list of pairs, while |= and update accept any iterable of pairs.

Every spelling merges the top level only. A key whose value is a dict is replaced whole by the later dict's value, so an override that mentions only sink.path drops sink.format.

The fix merges the nested level explicitly and puts that result last, so it beats the wholesale replacement from override. Configs nested 3 or 4 levels deep need a merge function that repeats this step at every level where both sides hold a dict.

An order event from an API arrives as a dict whose values are dicts and lists, several levels deep. Each bracket evaluates to an object and the next bracket applies to that object, so event["items"][1]["sku"] reads as: the items list, its second element, that element's sku.

Writes follow the same path: every bracket except the last reads an object, and the last one writes into it. The postcode lands in the inner address dict in place, and append grows the list the items key already holds. A write through a level that does not exist fails on the read before it, so event["billing"]["city"] = "York" raises KeyError: 'billing'.

A bracket path raises KeyError at the first missing level, and the message names only that key, not where it sat in the path. The standard guard chains .get(key, {}): a missing level returns an empty dict, and the next .get runs against that. It covers absent keys. It does not cover a key that is present with the value None, which is how JSON null arrives.

The default in .get("customer", {}) applies only when the key is absent. Here the key is present, so .get returns None and the next call fails. Writing (event.get("customer") or {}) treats absent and null alike, since None or {} evaluates to {}. Any other falsy value at that level, such as an empty string, also becomes {}, which is usually what code reading nested data wants.

Complete the loop so every event prints its city or unknown, including the event whose customer is null and the one with no address.

A dict of lists answers questions like which orders each customer placed. The loop needs a list under the key before it can append, and setdefault(key, []) supplies it: on a customer's first order it stores a new empty list and returns it, and on every later order it returns the list already stored. .append then grows that stored list in place.

Writing by_customer[customer] = [order_id] instead replaces the list on every order, and c-17 ends up holding only A-1003. The setdefault version works because a list is mutable, so the caller changes the stored object. A running total per key holds an int, which += on the returned value cannot change inside the dict, so totals use totals[key] = totals.get(key, 0) + amount. collections.defaultdict(list) does the same grouping job from the standard library.

job_config = DEFAULTS copies nothing. A variable is a name bound to an object, and that assignment binds a second name to the same dict, so a write through either name changes the single dict both see. job_config is DEFAULTS is True, and it stays True however many keys either name writes.

d.copy(), dict(d), {**d}, and the | and {**a, **b} merges from the merging section build a new outer dict; update and |= build nothing, since they write into the left dict. The new dict's key slots are its own, so assigning copy["retries"] = 0 leaves the original's slot alone. The objects in those slots are the same objects as in the original, so a nested list or dict is shared, and mutating it through either dict shows in both.

The run makes 3 kinds of write. alias["retries"] = 5 changes defaults because alias is defaults. shallow["retries"] = 0 leaves defaults at 5 because shallow has its own slots. shallow["tags"].append("pii") changes no slot at all: it mutates the list both slots point to, so the final is check prints True.

There are 2 fixes, and the choice depends on how much of the structure the code will change. copy.deepcopy(d) rebuilds every nested dict and list, so no object is shared. To survive cycles it records each object it copies in a memo dict and dispatches on every object's type in Python code, while .copy() is one pass in C over the top-level slots. On an order record with a few dozen fields across 2 or 3 nested levels, deepcopy typically runs tens to hundreds of times slower than .copy().

Worked out for the nightly job: at roughly 20 microseconds per deepcopy against well under 1 microsecond per .copy(), copying each of the 2 million events once adds about 40 seconds of CPU to every run. A pipeline that copies each record at 3 stages pays that 3 times, around 2 minutes. When code changes only one nested field, the cheaper fix is to copy that level and share the rest:

targeted owns its meta dict and still shares items with record, which is safe as long as nothing appends to items. Merged configs follow the same rule: DEFAULTS | job shares the nested sink dict with DEFAULTS, so config["sink"]["path"] = ... rewrites the defaults for every later job.

A dict comprehension builds a whole dict in one expression: {key_expr: value_expr for item in iterable if condition}. Python runs the for over the iterable, skips items where the optional if is false, evaluates the key and value expressions for each remaining item, and inserts the pair. The result is always a new dict, and the source is left as it was.

These 3 shapes cover most daily use: transform keys, filter pairs, transform values. An if ... else inside the value expression is a ternary and picks a value for every item; an if after the for is a filter and drops items. Putting if ... else after the for is a SyntaxError. Filtering this way is also how to remove keys from a dict: build the smaller dict instead of deleting while looping.

The source can be anything iterable. Pairing a CSV header with a row of values goes through zip, and indexing a list of records by id reads each record for both the key and the value.

A-1001 appears twice and the dict holds only refunded. A comprehension inserts pairs in iteration order, and a repeated key overwrites the earlier value while keeping its first slot. For a stream of change events that is deduplication keeping the latest state; anywhere else it is silent data loss. dict(zip(header, values)) builds the same record and is the shorter choice when nothing needs transforming or filtering.

Each item produces 1 assignment and the last one per key wins, so a comprehension cannot collect several items under a key; that stays a setdefault loop. It also cannot read the dict it is building or catch an exception per item, and when the logic needs either, a plain loop is clearer. Values it passes through untouched are the source's own objects, so {k: v for k, v in record.items()} shares nested lists with record the same way .copy() does.