Which Collections Allow Duplicates

Concepts covered: Sets

A duplicate is the same value appearing more than once. Lists and tuples store every copy and count every copy. A set stores one. A dict sits in between: keys are unique, values are free to repeat, and giving a dict the same key twice raises no error and keeps the last value.

Lists and tuples keep every copy

clicks = ["home", "cart", "home", "home"]
print(len(clicks))
clicks.remove("home")
print(clicks)

Output:

4
['cart', 'home', 'home']

The list took 4 items and len reported 4. remove deleted the first home only, so 2 copies remain. Each copy is a real item in its own slot. min counts them all, as do max and sum: sum([5, 5, 5]) is 15, and a tuple behaves the same way.

A set collapses copies

seen = set()
for user in [7, 7, 7, 3]:
    seen.add(user)
print(seen, len(seen))

Output:

{3, 7} 2

Of the 4 values added, 2 survive. add on a value already present does nothing and raises nothing, which is what makes a set the right place to record whether something has been seen rather than how often.

Dict keys: the last one wins

Write a dict literal with the same key twice and Python builds it left to right, so the second value replaces the first. The key keeps the position where it first appeared. The same rule governs dict() given a list of pairs, and plain assignment to a key that already exists.

config = {"retries": 3, "timeout": 30, "retries": 5}
print(config)
print(len(config))

pairs = [("a", 1), ("b", 2), ("a", 10)]
print(dict(pairs))

Output:

{'retries': 5, 'timeout': 30}
2
{'a': 10, 'b': 2}

Python raises nothing, and len reports 2. A config literal with a repeated key silently discards the earlier value, and the mistake only surfaces when the program behaves as if retries were 5. The pairs example keeps a in front with the value 10 from the later pair.

Using the rule on purpose: counting

Because keys are unique, a dict is the natural place to count how often each item of a list appears: the item becomes the key and its count the value. in checks whether the key exists before deciding whether to start at 1 or add 1 to the count already there.

clicks = ["home", "cart", "home", "home", "cart"]
counts = {}

for page in clicks:
    if page in counts:
        counts[page] = counts[page] + 1
    else:
        counts[page] = 1

print(counts)
print("distinct pages:", len(counts))
print("total clicks:", len(clicks))

{'home': 3, 'cart': 2}
distinct pages: 2
total clicks: 5

The list reports 5 because it kept every click. The dict reports 2 because home and cart are the only keys that ever existed, no matter how many times each was seen.

Practice

The first exercise mixes a set literal and a dict literal with repeats. The second has you write the counting pattern.

What does this program print?

tags = {"b", "a", "b", "c", "a"}
counts = {"a": 1, "b": 2, "a": 3}
print(len(tags) + len(counts), counts["a"])

• 7 3
• 5 1
• 3 3
• 5 3

Trace the counting loop on 3 clicks and the 2 branches separate cleanly. A page's first visit finds no key, so the count starts at 1, and the key takes its position in the dict at that moment. Every later visit finds the key and adds 1 to the value under it, leaving the position alone. Flip the test to not in and the branches swap jobs with it: the first branch starts the count and the else adds 1.

About this lesson

This section is part of the Data Structures: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.