Copying Dictionaries

Concepts covered: Copying Lists

job_config = DEFAULTS copies nothing. A variable is a name bound to an object, and that assignment binds a second name to the same dict, so a write through either name changes the single dict both see. job_config is DEFAULTS is True, and it stays True however many keys either name writes.

Shallow copies: new slots, shared values

d.copy(), dict(d), {**d}, and the | and {**a, **b} merges from the merging section build a new outer dict; update and |= build nothing, since they write into the left dict. The new dict's key slots are its own, so assigning copy["retries"] = 0 leaves the original's slot alone. The objects in those slots are the same objects as in the original, so a nested list or dict is shared, and mutating it through either dict shows in both.

defaults = {"retries": 3, "tags": ["orders"]}
alias = defaults
alias["retries"] = 5

shallow = defaults.copy()
shallow["retries"] = 0
shallow["tags"].append("pii")

print(defaults)
print(shallow)
print(shallow["tags"] is defaults["tags"])
{'retries': 5, 'tags': ['orders', 'pii']}
{'retries': 0, 'tags': ['orders', 'pii']}
True

The run makes 3 kinds of write. alias["retries"] = 5 changes defaults because alias is defaults. shallow["retries"] = 0 leaves defaults at 5 because shallow has its own slots. shallow["tags"].append("pii") changes no slot at all: it mutates the list both slots point to, so the final is check prints True.

.copy() (shallow)
  • New outer dict with its own key slots
  • Nested lists and dicts shared with the source
  • Cost grows with the number of top-level keys
  • Right for flat records and nested data nobody mutates
copy.deepcopy (deep)
  • New container at every level
  • Nothing shared at any depth
  • Visits every nested object and records each in a memo dict
  • Right when code mutates nested data it does not own

Getting a copy with nothing shared

There are 2 fixes, and the choice depends on how much of the structure the code will change. copy.deepcopy(d) rebuilds every nested dict and list, so no object is shared. To survive cycles it records each object it copies in a memo dict and dispatches on every object's type in Python code, while .copy() is one pass in C over the top-level slots. On an order record with a few dozen fields across 2 or 3 nested levels, deepcopy typically runs tens to hundreds of times slower than .copy().

Worked out for the nightly job: at roughly 20 microseconds per deepcopy against well under 1 microsecond per .copy(), copying each of the 2 million events once adds about 40 seconds of CPU to every run. A pipeline that copies each record at 3 stages pays that 3 times, around 2 minutes. When code changes only one nested field, the cheaper fix is to copy that level and share the rest:

import copy

record = {"order_id": "A-1001", "meta": {"source": "web"}, "items": [{"sku": "SKU-4"}]}

full = copy.deepcopy(record)
targeted = {**record, "meta": record["meta"].copy()}

targeted["meta"]["replayed"] = True
print(record["meta"], full["items"] is record["items"])

targeted owns its meta dict and still shares items with record, which is safe as long as nothing appends to items. Merged configs follow the same rule: DEFAULTS | job shares the nested sink dict with DEFAULTS, so config["sink"]["path"] = ... rewrites the defaults for every later job.

Copy at the boundary

Functions are where shared state bites. A dict passed as an argument is the caller's object, so a function that adds a field to it, or to anything nested inside it, edits the caller's data. A function that returns a modified record should copy each level it changes before changing it. The function below tags replayed events and was meant to leave its input alone.

About this lesson

This section is part of the Dictionaries: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.