defaultdict Usage

Concepts covered: Collections

Grouping rows under a key runs through most aggregation code: orders by customer, files by date, errors by service. With a plain dict, every append needs a check that the key's list exists yet. defaultdict moves that check inside the dict. You give it a factory, and a lookup on a missing key calls the factory, stores the result under that key and returns it.

orders = [("ana", 30), ("ben", 12), ("ana", 45), ("cy", 8), ("ben", 20)]

by_customer = {}
for customer, amount in orders:
    if customer not in by_customer:
        by_customer[customer] = []
    by_customer[customer].append(amount)

by_customer = {}
for customer, amount in orders:
    by_customer.setdefault(customer, []).append(amount)

The same grouping with defaultdict:

from collections import defaultdict

orders = [("ana", 30), ("ben", 12), ("ana", 45), ("cy", 8), ("ben", 20)]

by_customer = defaultdict(list)
for customer, amount in orders:
    by_customer[customer].append(amount)

print(dict(by_customer))

Output:

{'ana': [30, 45], 'ben': [12, 20], 'cy': [8]}

You pass list, the type itself, with no parentheses. The dict calls it with no arguments each time it meets a new key, so every customer gets a fresh list of their own. Converting with dict() before printing gives a plain dict, which also goes back to raising KeyError on a mistyped key.

Any zero-argument callable is a factory

int() returns 0, so defaultdict(int) is a tally you bump with += 1. set collects distinct values. A function you define works too, provided it takes no arguments. As the 2 dicts below fill, watch when each key first appears in each.

from collections import defaultdict

events = [("eu", "click"), ("us", "view"), ("eu", "view"),
          ("ap", "click"), ("us", "click")]

by_region = defaultdict(list)
clicks = defaultdict(int)
for region, action in events:
    by_region[region].append(action)
    if action == "click":
        clicks[region] += 1

print(dict(by_region))
print(dict(clicks))
{'eu': ['click', 'view'], 'us': ['view', 'click'], 'ap': ['click']}
{'eu': 1, 'ap': 1, 'us': 1}

'us' joined by_region on the second event and clicks only on the fifth, the first time anything looked up clicks['us']. A key comes into existence at its first lookup. When each value is a small record, write the factory as a function:

from collections import defaultdict

def new_stats():
    return {"rows": 0, "bytes": 0}

stats = defaultdict(new_stats)
for table, size in [("orders", 500), ("users", 120), ("orders", 300)]:
    stats[table]["rows"] += 1
    stats[table]["bytes"] += size

print(dict(stats))

Output:

{'orders': {'rows': 2, 'bytes': 800}, 'users': {'rows': 1, 'bytes': 120}}

The lookup that writes

That rule has a side effect you must predict: d[key] on a missing key inserts it, even inside an if that was only meant to look. .get() and in never call the factory, so they are the safe ways to ask about a key.

from collections import defaultdict

by_customer = defaultdict(list)
by_customer["ana"].append(30)

if by_customer["dan"]:
    print("dan has orders")
print(len(by_customer), list(by_customer))
print(by_customer.get("eve"), len(by_customer))
print("fay" in by_customer, len(by_customer))

Output:

2 ['ana', 'dan']
None 2
False 2

The if saw an empty list and printed nothing, yet 'dan' now sits in the dict with no orders, and a report counting customers would count dan. Converting to dict() once grouping ends shuts this door, because later lookups go back to raising KeyError.

Do
  • Pass the factory itself: defaultdict(list)
  • Use in or .get() when you only want to look
  • Convert with dict(d) once grouping is finished
  • Use Counter over defaultdict(int) when you need top items
Avoid
  • Write defaultdict([]): a list can't be called, so it raises TypeError
  • Test if d[key]: on a defaultdict
  • Nest with a call: defaultdict(defaultdict(int)) raises TypeError
  • Expect a missing key to raise KeyError as a plain dict would

A defaultdict keeps its factory in an attribute, default_factory. Setting d.default_factory = None once grouping ends makes missing keys raise KeyError again, and d stays the same object. That helps when other code already holds a reference to it, which a dict() copy would leave behind. .get(key, 0) takes a fallback of your own. Like plain .get(), it never calls the factory and stores nothing.

What does this program print?

from collections import defaultdict

d = defaultdict(int)
d["a"] += 2
total = d["a"] + d["b"]
print(total, len(d), d.get("c"))

About this lesson

This section is part of the Collections: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.