Dictionary Gotchas

Concepts covered: Dict Iteration

Each trap below runs without an error on at least some inputs, which is how it survives into production code, so predict each output before you open it.

Changing size while looping

A dict iterator notices when the dict gains or loses keys and raises RuntimeError on its next step. The error arrives after the first deletion has already happened, so the dict is left half cleaned:

sessions = {"s1": 0, "s2": 45, "s3": 0, "s4": 12}

try:
    for sid in sessions:
        if sessions[sid] == 0:
            del sessions[sid]
except RuntimeError as err:
    print("RuntimeError:", err)
print(sessions)

for sid in list(sessions):
    if sessions[sid] == 0:
        del sessions[sid]
print(sessions)

Output:

RuntimeError: dictionary changed size during iteration
{'s2': 45, 's3': 0, 's4': 12}
{'s2': 45, 's4': 12}

list(sessions) is a snapshot of the keys taken before any deletion, so the loop never walks the dict it changes. Building a new dict works too: {k: v for k, v in sessions.items() if v != 0}. Assigning a new value to an existing key keeps the size the same and is allowed mid-loop.

One list shared by every key

dict.fromkeys(keys, value) stores the same value object under every key. With an int or a string that is harmless. With a list or a dict, every key points at 1 shared object:

teams = dict.fromkeys(["web", "data"], [])
teams["web"].append("ana")
print(teams)

teams = {name: [] for name in ["web", "data"]}
teams["web"].append("ana")
print(teams)

Output:

{'web': ['ana'], 'data': ['ana']}
{'web': ['ana'], 'data': []}

The comprehension evaluates [] once per key, so each team gets its own list.

What does this program print?

limits = dict.fromkeys(["free", "pro"], {"seats": 1})
limits["pro"]["seats"] = 10
print(limits["free"]["seats"], len(limits))

Falsy values that look missing

d.get(k) returns None for a missing key, so if not d.get(k) fires for a missing key. It also fires for a stored 0 or "", and for [] and False too. When a falsy value is real data, test for the key with in:

stock = {"mug": 0, "tee": 8}

for item in ["mug", "hat"]:
    if not stock.get(item):
        print("get says missing:", item)

for item in ["mug", "hat"]:
    if item not in stock:
        print("in says missing:", item)

Output:

get says missing: mug
get says missing: hat
in says missing: hat

Updates keep a key's position

Assigning to a key that already exists leaves it where it was. Deleting it and adding it back moves it to the end. Equality ignores order, so 2 dicts can be equal and still loop in different orders:

steps = {"a": 1, "b": 2, "c": 3}
steps["a"] = 10
print(list(steps))

del steps["a"]
steps["a"] = 10
print(list(steps))
print(steps == {"a": 10, "b": 2, "c": 3})

Output:

['a', 'b', 'c']
['b', 'c', 'a']
True

A key that changes after it is stored

A dict files each key under the hash it had when it went in. Reassign a field that __hash__ reads, and the key stays filed under its old hash. A lookup by the new value goes to the wrong slot, and a lookup by the old value finds a key that no longer compares equal:

class Ticket:
    def __init__(self, number):
        self.number = number

    def __eq__(self, other):
        return isinstance(other, Ticket) and self.number == other.number

    def __hash__(self):
        return hash(self.number)

t = Ticket(1)
owners = {t: "ana"}
t.number = 2
print(Ticket(1) in owners, Ticket(2) in owners, t in owners)
print(len(owners), [k.number for k in owners])

Output:

False False False
1 [2]

The entry is still in the dict and still counted, yet no lookup can reach it. Keep keys to strings and ints, or to objects whose hashed fields you never reassign.

Do
  • Loop over list(d) or build a new dict when the loop deletes keys
  • Build per-key lists with {k: [] for k in keys}
  • Test k in d when 0, "" or None are valid values
  • Hash only fields that never change after the object becomes a key
Avoid
  • Delete or add keys while iterating over the dict itself
  • Pass a list or dict as the value to dict.fromkeys
  • Use if not d.get(k) to mean the key is absent
  • Expect reassigning a key to move it to the end of the order

About this lesson

This section is part of the Dictionaries: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.