Union: Combining Sets

Concepts covered: Set Operations

Pipelines rarely get their keys from a single place. Say Monday's purchases arrive on 2 event feeds, the website and the mobile app, and each feed is loaded into a set of buyer IDs. A customer who bought on both shows up in both sets. Finance wants the number of distinct people who bought on Monday, and that number comes from the union: every ID present in at least 1 of the sets, each counted once.

The | operator and the union() method

Python offers union in 2 spellings. web | app reads as "in web or in app" and builds a new set. web.union(app) is the method spelling and builds an equal set. Neither form changes web or app: each keeps the IDs it started with, which matters when a later step still needs the per-feed sets, for a per-channel breakdown say.

Python prints a set in its internal storage order, and for strings that order can change between runs. The examples pass each set to sorted(), which returns a list of the items in ascending order, so every printout comes out the same.

web = {101, 102, 103, 104}
app = {103, 104, 105}

everyone = web | app
also_everyone = web.union(app)

print(sorted(everyone))
print(everyone == also_everyone)
print(len(web), len(app), len(everyone))
print(sorted(web))

Output:

[101, 102, 103, 104, 105]
True
4 3 5
[101, 102, 103, 104]

The third printed line holds the lesson about counts. The feeds hold 4 and 3 IDs, 7 in total, yet the union holds 5, because 103 and 104 appear once each. Adding the sizes of 2 sets overcounts whenever they overlap, and summing per-feed counts is a common way for a dashboard to report more customers than exist. The size of that overcount is the overlap, which the next section computes directly. The last line shows web still holding its original 4 IDs.

What union() accepts that | refuses

The operator insists on a set on both sides: web | [106] raises TypeError. The method takes any iterable, meaning a list, a string, a range or a dict (whose keys it uses), and converts the input as it reads it. It also takes several arguments, merging all of them into a single new set.

web = {101, 102, 103}
batch = [103, 103, 106, 107]

merged = web.union(batch)
print(sorted(merged))

wide = web.union([108], range(109, 111), {101})
print(sorted(wide))

Output:

[101, 102, 103, 106, 107]
[101, 102, 103, 108, 109, 110]

The list carried 103 twice. The result kept a single 103, which web already held, and added 106 and 107. The second call merged a list, a range and another set at once, and 101, already in web, appeared a single time.

The | operator

• Both sides must be sets, or Python raises TypeError
• Chains to more inputs: a | b | c
• Short enough to sit inside a larger expression

The union() method

• Takes lists, strings, ranges and dicts as well as sets
• Merges any number of inputs in a single call
• Converts its inputs, so no set(...) wrapper is needed

Folding a list of sets

Daily partitions often arrive as a list holding a set per day. Start from an empty set and fold each day in with |. The empty set contributes nothing, so after the first step seen holds the first day's IDs, and every later step adds whatever that day brings that is new.

partitions = [{1, 2}, {2, 3}, {3, 4, 5}]

seen = set()
for ids in partitions:
    seen = seen | ids
    print(sorted(seen))

Output:

[1, 2]
[1, 2, 3]
[1, 2, 3, 4, 5]

A job's tags live in a set, and a new tag arrives as a plain string. What does this program print?

tags = {"etl", "daily"}
tags = tags.union("sql")
print(sorted(tags))
print(len(tags))

• ['daily', 'etl', 'sql'] then 3
• ['daily', 'etl', 'l', 'q', 's'] then 5
• TypeError, since union() needs a set argument
• ['daily', 'etl'] then 2, since tags is unchanged

Adding a whole string as 1 item

To add a word as 1 tag, hand union() a container holding it: the list ["sql"] yields the whole string, and so does tags | {"sql"}. Each argument is read on its own, so 1 call can mix a correct input with a careless one, as the third line shows: "hourly" arrives whole and "sql" splits into letters.

tags = {"etl", "daily"}
print(sorted(tags.union(["sql"])))
print(sorted(tags | {"sql"}))
print(sorted(tags.union("sql", ["hourly"])))

Output:

['daily', 'etl', 'sql']
['daily', 'etl', 'sql']
['daily', 'etl', 'hourly', 'l', 'q', 's']

Each argument to union() is looped over on its own, so a list contributes whole IDs while a bare string contributes characters. A repeated ID in any argument lands in the result once, and the calling set keeps its contents.

About this lesson

This section is part of the Sets: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.