Converting Between Dicts and JSON

Concepts covered: Dictionaries, JSON Handling

JSON is a text format for data, and Python's json module converts between it and dicts. json.dumps(d) returns a string, and json.loads(text) parses a string back into Python values. Each JSON type has a Python counterpart:

import json

order = {"id": 7, "paid": True, "coupon": None, "items": ["mug", "tee"], "total": 19.5}
text = json.dumps(order)
print(text)
print(type(text).__name__)
back = json.loads(text)
print(back == order)

Output:

{"id": 7, "paid": true, "coupon": null, "items": ["mug", "tee"], "total": 19.5}
str
True

The output is a str in JSON spelling, with lowercase true and null for None. Every string sits in double quotes. Loading it gives a dict equal to the original, because every value here has a JSON counterpart and every key was already a string.

Keys come back as strings

JSON object keys can only be strings, so dumps converts int and float keys to text. Bool and None keys get the same treatment, and loads has no record of what any of them were. A dict keyed by product ID changes shape on the round trip:

import json

stock = {101: 4, 102: 0}
text = json.dumps(stock)
print(text)
back = json.loads(text)
print(back)
print(back == stock, back.get(101), back.get("101"))

Output:

{"101": 4, "102": 0}
{'101': 4, '102': 0}
False None 4

The loaded dict is unequal to the original, and a lookup by the int 101 misses. Convert the keys back where you load: {int(k): v for k, v in back.items()}. Worse, 2 different keys can turn into the same string. The equal-key collapse happens once inside the dict, then JSON merges what was left:

import json

mixed = {1: "int key", "1": "str key", True: "bool key"}
print(len(mixed))
text = json.dumps(mixed)
print(text)
print(json.loads(text))

Output:

2
{"1": "bool key", "1": "str key"}
{'1': 'str key'}

True merged into 1 inside the dict, leaving 2 keys. dumps then wrote both as "1", producing JSON with a repeated key, and loads kept the last value it read for it. The whole round trip ran without raising an error.

What does this program print?

import json

counts = {2024: 10, "2024": 3}
restored = json.loads(json.dumps(counts))
print(restored)

• {2024: 10, '2024': 3}
• {'2024': 10, '2024': 3}
• {'2024': 13}
• {'2024': 3}

Values JSON cannot hold

Dates and datetimes have no JSON type, and neither do bytes or objects of your own classes, so dumps raises TypeError at the first one it meets. Its default argument takes a function that dumps calls on each value it cannot write, and the function's return value is written in its place. Floats fail the other way: dumps writes NaN and infinity as NaN and Infinity, which strict JSON parsers in other languages reject, so pass allow_nan=False to fail at the source.

import json
from datetime import date

event = {"user": "u3", "day": date(2024, 3, 1)}
try:
    json.dumps(event)
except TypeError as err:
    print("TypeError:", err)

print(json.dumps(event, default=str))

ratio = {"ratio": float("nan")}
print(json.dumps(ratio))
try:
    json.dumps(ratio, allow_nan=False)
except ValueError:
    print("ValueError: NaN refused")

Output:

TypeError: Object of type date is not JSON serializable
{"user": "u3", "day": "2024-03-01"}
{"ratio": NaN}
ValueError: NaN refused

default=str keeps the program running, but loading that text gives back the string "2024-03-01", never a date. The code reading the JSON has to know which fields to convert back.

Stable, readable text

Equal dicts can produce different JSON text, because dumps writes keys in insertion order while dict equality ignores order. sort_keys=True fixes the order, which matters when you compare outputs as text or store them where a changed byte means a changed file. indent=2 lays the text out for people to read:

import json

a = {"x": 1, "y": 2}
b = {"y": 2, "x": 1}
print(a == b, json.dumps(a) == json.dumps(b))
print(json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True))

report = {"region": "eu", "counts": {"ok": 12, "failed": 1}}
print(json.dumps(report, indent=2, sort_keys=True))

Output:

True False
True
{
  "counts": {
    "failed": 1,
    "ok": 12
  },
  "region": "eu"
}

About this lesson

This section is part of the Dictionaries: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.