Set Difference

Concepts covered: Set Operations

Difference answers a one-sided question: which elements of the left set are absent from the right set. today - yesterday keeps every ID active today that was not active yesterday. Anything in yesterday that today lacks is ignored, because the result only ever holds elements of the left operand. Flip the operands and the question flips with them.

yesterday = {11, 12, 13, 14}
today = {13, 14, 15}

new_users = today - yesterday
gone_users = yesterday - today

print(sorted(new_users))
print(sorted(gone_users))
print(today - yesterday == yesterday - today)

Output:

[15]
[11, 12]
False

The same 2 sets answer 2 different business questions. today - yesterday gives the new users a growth report counts, and yesterday - today gives the users who went quiet, the input to a churn alert. The False on the last line is the reason to name each result after its direction, as new_users and gone_users do.

Syncing a table: inserts and deletes

A nightly job keeps a warehouse table in step with a source table. Comparing the 2 sets of primary keys splits the work 3 ways. Keys in the source but not the warehouse belong to new rows to insert. Keys in the warehouse but not the source were deleted upstream. Keys in both need a value check for updates, and that group is an intersection.

source = {"A1", "A2", "A3", "A5"}
warehouse = {"A1", "A2", "A4"}

to_insert = source - warehouse
to_delete = warehouse - source
to_compare = source & warehouse

print("insert", sorted(to_insert))
print("delete", sorted(to_delete))
print("compare", sorted(to_compare))

Output:

insert ['A3', 'A5']
delete ['A4']
compare ['A1', 'A2']

Removing several groups at once

a.difference(b, c) removes everything found in b or in c, and it takes a list wherever the operator would demand a set. The operator form chains: a - b - c runs left to right, so Python computes (a - b) - c. Both drop the same elements from a.

signups = {1, 2, 3, 4, 5, 6}
bots = [2, 6]
test_accounts = {3}

real = signups.difference(bots, test_accounts)
print(sorted(real))
print(sorted(signups - set(bots) - test_accounts))
print(sorted(signups))

Output:

[1, 4, 5]
[1, 4, 5]
[1, 2, 3, 4, 5, 6]

The operator line needed set(bots) because - refuses a list. Subtracting an ID the left set lacks is harmless: {1, 2} - {99} returns a set holding 1 and 2. That makes difference the safer tool for a cleanup list that may name IDs already gone, where calling remove(99) would raise KeyError. And signups printed in full on the last line: the method built a new set and left the original alone.

Do

• Name each result after its direction: new_users = today - yesterday
• Use difference() when some of the inputs are lists
• Add parentheses whenever the grouping matters: a - (b - c)

Avoid

• Don't treat a - b and b - a as interchangeable
• Don't loop remove() over IDs that may already be gone
• Don't subtract lists: [1, 2] - [1] raises TypeError

Both lines subtract the same 3 sets. What does the program print?

a = {1, 2, 3}
b = {2, 3}
c = {3}

print(sorted(a - b - c))
print(sorted(a - (b - c)))

• [1] then [1, 3]
• [1] then [1]
• [1, 3] then [1]
• [] then [1]

A reversed difference raises no error. The result holds only elements of the left operand, so when the left side is a short group of exceptions that all sit in the right side, the answer is an empty set on every run, printed as a clean [] and a count of 0.

active = {21, 22, 23, 24}
paused = {22, 24}

print(sorted(active - paused))
print(sorted(paused - active), len(paused - active))

Output:

[21, 23]
[] 0

Put the set you keep on the left and read the line as "keep these, except those".

About this lesson

This section is part of the Sets: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.