Set Intersection

Concepts covered: Set Operations

Intersection keeps what the sets have in common: an element survives only if every operand contains it. Back in the Blend, the artists Maya and Theo both play form the intersection of their 2 artist sets. Python writes it maya & theo, read as "in maya and in theo", or maya.intersection(theo).

maya = {"Adele", "Bonobo", "Caribou", "Daft Punk"}
theo = {"Bonobo", "Daft Punk", "Earl Sweatshirt"}

shared = maya & theo
print(sorted(shared))
print(sorted(theo.intersection(maya)))
print(len(maya) + len(theo) - len(shared))
print(len(maya | theo))

Output:

['Bonobo', 'Daft Punk']
['Bonobo', 'Daft Punk']
5
5

Swapping the operands changed nothing, since a & b and b & a hold the same elements. The last 2 lines settle the overcount from the union section: 4 artists plus 3 artists minus the 2 they share equals the 5 in the union. For 2 sources, the gap between the summed counts and the distinct count is exactly the size of their overlap. With 3 or more sources, an ID present in all 3 is counted 3 times, so the gap is no longer the size of any single overlap.

When nothing is shared

An intersection with no common elements is still a set, an empty one, and Python prints it as set(). The braces {} already mean an empty dict, so an empty set needs a name of its own. Comparing with overlap == set() or testing len(overlap) == 0 reads that result correctly.

nightly = {201, 202}
weekly = {301, 302}

overlap = nightly & weekly
print(overlap)
print(len(overlap))
print(overlap == set())

Output:

set()
0
True

Users active on every day

A retention report asks which users were active on all 3 days of a period. Both forms accept more than 2 sets: mon & tue & wed, or mon.intersection(tue, wed). When the days arrive as a list, a loop does the same job, and its starting value decides whether the answer is right. Step through it as the set narrows day by day.

days = [
    {101, 102, 103, 105},
    {102, 103, 105, 107},
    {101, 103, 105, 108},
]
common = days[0]
for day in days[1:]:
    common = common & day
    print(sorted(common))
print(len(common))

[102, 103, 105]
[103, 105]
2

The loop starts from days[0] and each & can only remove IDs, so the set shrinks from 4 to 3 to 2. Starting from set() instead, the habit that serves union well, empties the result on the first step: an empty set shares nothing with anything. Union and intersection want opposite starting values.

Checking incoming keys against loaded ones

The method form of intersection also takes a list, so a batch of incoming keys can be checked against the loaded keys without converting the batch first. Repeats in the batch collapse, because the result is a set.

loaded = {5001, 5002, 5003, 5004}
incoming = [5003, 5003, 5005, 5001]

already_there = loaded.intersection(incoming)
print(sorted(already_there))
print(len(incoming), len(already_there))

Output:

[5001, 5003]
4 2

Of the 4 incoming entries, 2 distinct IDs were already loaded: 5001 and 5003, with the repeated 5003 counted once. A load step that skips those keys avoids writing duplicate rows, and 5005, absent from loaded, never appears in the result.

A retention job looks for IDs active on all 3 days. What does it print?

mon = {1, 2, 3}
tue = {2, 3, 4}
wed = {3, 4, 5}

every_day = set()
for day in [mon, tue, wed]:
    every_day = every_day & day
print(every_day)

• {3}, the single ID present on all 3 days
• {1, 2, 3, 4, 5}, every ID seen on any day
• {}, the empty result printed with braces
• set(), because the loop starts from an empty set

The repair changes 1 line: start from the first set in the list. A period of a single day still works. Its slice [1:] is empty, the loop body never runs, and the answer is that day's set, which is correct: everyone active that day was active on every day of the period.

days = [{1, 2, 3}, {2, 3, 4}, {3, 4, 5}]
every_day = days[0]
for day in days[1:]:
    every_day = every_day & day
print(sorted(every_day), len(every_day))

one_day = [{7, 8}]
result = one_day[0]
for day in one_day[1:]:
    result = result & day
print(sorted(result))

Output:

[3] 1
[7, 8]

The pattern to carry forward: start from the first set, loop over the rest with &, and expect the result to shrink or hold steady. Its length is the number a retention report shows.

About this lesson

This section is part of the Sets: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.