Symmetric Difference

Concepts covered: Set Operations

Symmetric difference keeps the elements that sit in exactly 1 of 2 sets and drops the ones they share. Reconciliation is where it earns its keep: an orders service and a billing service each hold a set of order IDs for the day, and every ID present in only 1 of them is a record someone has to chase. Python writes it orders ^ billing.

orders = {7001, 7002, 7003, 7004}
billing = {7002, 7003, 7005}

mismatched = orders ^ billing
print(sorted(mismatched))
print(mismatched == ((orders - billing) | (billing - orders)))
print(mismatched == ((orders | billing) - (orders & billing)))
print(sorted(billing.symmetric_difference(orders)))

Output:

[7001, 7004, 7005]
True
True
[7001, 7004, 7005]

The 2 True lines show what ^ is made of: the 2 one-sided differences joined together, or equally, the union with the intersection taken out. Operand order does not matter here, as the last line confirms with billing on the left.

The result says 7001, 7004 and 7005 disagree, and nothing about which system holds each. If the fix depends on the side, such as re-sending an order to billing versus voiding a charge, compute orders - billing and billing - orders separately. Use ^ when the question is "do these match, and which IDs differ": an empty result means the 2 sets hold exactly the same elements.

Chaining ^ over 3 sets

Applying ^ across 3 sets looks as if it should give the elements found in exactly 1 of them. Follow the ID 2, which sits in all 3 sets, through each step.

a = {1, 2}
b = {2, 3}
c = {2, 4}

step = a ^ b
print(sorted(step))
step = step ^ c
print(sorted(step))
print(2 in a, 2 in b, 2 in c)

[1, 3]
[1, 2, 3, 4]
True True True

The first ^ drops 2 because a and b share it. The second ^ compares that result with c, which holds 2 while the result does not, so 2 comes back. Each ^ flips an element's presence, so a chain keeps the elements that appear in an odd number of the sets: 1 set, 3 sets, 5 sets all qualify. For elements in exactly 1 set, take away everything that appears in 2 or more.

a = {1, 2}
b = {2, 3}
c = {2, 4}

in_two_or_more = (a & b) | (a & c) | (b & c)
exactly_one = (a | b | c) - in_two_or_more
print(sorted(exactly_one))
print(sorted(a ^ b ^ c))

Output:

[1, 3, 4]
[1, 2, 3, 4]

Choosing ^ or symmetric_difference()

The ^ operator

• Both operands must be sets
• Chains, keeping elements with an odd count
• Fits inside a longer expression, such as a ^ b == set()

symmetric_difference()

• Accepts a list, string or range as its argument
• Takes exactly 1 argument, where union() takes many
• Its name tells a reader what the line computes

expected_cols = {"id", "email", "created_at"}
actual_cols = ["id", "email", "signup_ts"]

drift = expected_cols.symmetric_difference(actual_cols)
print(sorted(drift))
print(len(drift) == 0)

Output:

['created_at', 'signup_ts']
False

The method compared a set with the list of column names that arrived in the data, and the result named both sides of the drift: a column the contract expects and a column nobody asked for. The False is the alert condition. For 2 sets in a short expression ^ is the compact choice; when the input is a list, or the line should explain itself to whoever reviews it, use the method.

A job chains ^ across 3 daily snapshots of IDs. What does it print?

mon = {1, 2, 3}
tue = {2, 3}
wed = {3, 4}
print(sorted(mon ^ tue ^ wed))

• [1, 4]
• [1, 2, 3, 4]
• [1, 3, 4]
• [3]

To test whether 2 collections match, compare 2 sets. A set never equals a list, so shelf == scanned below is False even though both name the same SKUs. The symmetric difference reads the list as a set, so the repeated S1 collapses and the result is empty.

shelf = {"S1", "S2", "S3"}
scanned = ["S3", "S1", "S2", "S1"]

print(sorted(shelf.symmetric_difference(scanned)))
print(shelf == scanned)
print(shelf == set(scanned))

Output:

[]
False
True

Reach for the method when one side arrives as a list: it converts the list, drops repeats and returns what sits on only 1 side. An empty result means the 2 collections match.

About this lesson

This section is part of the Sets: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.