Merging Dictionaries

Concepts covered: Dict Merging

A pipeline job's settings usually arrive in layers: package defaults, then overrides for the environment, then whatever this run passed on the command line. Combining them applies one rule over and over: when 2 dicts share a key, the dict merged later wins. Python has 4 spellings of that rule, and they split on whether the left-hand dict is changed.

SpellingGives backChanges the left dictPython
`a.update(b)`None; `a` holds the resultYesAny
`a |= b``a`, holding the resultYes3.9+
`a | b`A new dictNo3.9+
`{**a, **b}`A new dictNo3.5+
DEFAULTS = {"batch_size": 500, "retries": 3, "timeout_s": 30}
env = {"retries": 5}
cli = {"batch_size": 1000}

config = DEFAULTS | env | cli
print(config)
print(DEFAULTS)

Read the chain left to right as lowest priority to highest: the layer that must win goes last. Key position follows a separate rule. A key keeps the slot of its first appearance and only its value changes, so {"a": 1, "b": 2} | {"b": 9, "a": 7} gives {'a': 7, 'b': 9}, in the left dict's order. Code that writes merged dicts out as CSV rows gets its column order from the defaults.

Why in-place merges leak

A module-level DEFAULTS lives as long as the process. A runner that loops over 3 jobs and calls DEFAULTS.update(job_overrides) leaves job 1's retries: 0 in place when job 2 builds its config, and job 2 never asked for it. Tests that run one job per process never catch this. | and {**a, **b} build a fresh dict on every call, so the shared base never changes. update and |= belong on dicts the current code owns, such as one it built a line earlier.

{**a, **b} also accepts literal keys after the unpacks: {**DEFAULTS, **env, "run_id": run_id} stamps a value no earlier layer can override. | requires a dict on both sides and raises TypeError for a list of pairs, while |= and update accept any iterable of pairs.

The runner below should let each job's settings beat the defaults. Run it: the job asked for 0 retries and a 120-second timeout, and the printed config disagrees.

Merges are one level deep

Every spelling merges the top level only. A key whose value is a dict is replaced whole by the later dict's value, so an override that mentions only sink.path drops sink.format.

DEFAULTS = {"retries": 3, "sink": {"format": "parquet", "path": "s3://lake/orders"}}
override = {"sink": {"path": "s3://lake/orders_v2"}}

print(DEFAULTS | override)
fixed = DEFAULTS | override | {"sink": DEFAULTS["sink"] | override.get("sink", {})}
print(fixed)

The fix merges the nested level explicitly and puts that result last, so it beats the wholesale replacement from override. Configs nested 3 or 4 levels deep need a merge function that repeats this step at every level where both sides hold a dict.

With the order settled, one choice remains: which spelling the runner uses for each job.

A runner builds configs for 3 jobs in a row from a module-level DEFAULTS dict. Which line applies each job's overrides on top of the defaults without job 1's settings reaching job 2?

About this lesson

This section is part of the Dictionaries: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.