Using zip() for Combining

Concepts covered: Parallel Iteration

You have walked 2 lists side by side with zip() in a for loop. Called on its own, zip() returns a zip object that yields tuples and can be consumed once. Whatever you pass it to sets the final shape: list() gives pairs and dict() turns those pairs into a mapping.

job_ids = ["load_orders", "dedupe_users", "build_daily"]
minutes = [42, 7, 19]
print(list(zip(job_ids, minutes)))
print(dict(zip(job_ids, minutes)))

Output:

[('load_orders', 42), ('dedupe_users', 7), ('build_daily', 19)]
{'load_orders': 42, 'dedupe_users': 7, 'build_daily': 19}

zip() accepts any number of iterables and builds a tuple with 1 item from each, so 3 parallel columns become 3-field rows that a for loop can unpack in a single line:

job_ids = ["load_orders", "dedupe_users", "build_daily"]
minutes = [42, 7, 19]
statuses = ["ok", "failed", "ok"]
for job, mins, status in zip(job_ids, minutes, statuses):
    print(job, mins, status)

Output:

load_orders 42 ok
dedupe_users 7 failed
build_daily 19 ok

dict(zip(...)) has its own silent failure. When a key repeats, each later pair overwrites the earlier value, the same as assigning to that key twice:

job_ids = ["load_orders", "build_daily", "load_orders"]
minutes = [42, 19, 45]
runtimes = dict(zip(job_ids, minutes))
print(runtimes)
print(len(job_ids), len(runtimes))

Output:

{'load_orders': 45, 'build_daily': 19}
3 2

The first load_orders runtime of 42 is gone. Without any error, 3 job entries became 2 dict entries. If the key column can repeat, compare len(job_ids) with the length of the dict you built, as the last line does, before trusting the mapping.

Turning rows into records

Extracts often arrive as a header row plus value rows, the layout of a CSV file or a query result. dict(zip(header, row)) labels each value with its column name, and wrapping that in a function lets map() apply it to every row.

header = ["id", "email", "country"]
rows = [
    [1, "ana@site.com", "PT"],
    [2, "bo@site.com", "BR"],
]


def to_record(row):
    return dict(zip(header, row))


for record in map(to_record, rows):
    print(record)

Output:

{'id': 1, 'email': 'ana@site.com', 'country': 'PT'}
{'id': 2, 'email': 'bo@site.com', 'country': 'BR'}

The shortest input sets the length

zip() stops as soon as any input is exhausted, and it raises no error about the leftovers. Step through a mismatch where the runtime list lost its last value somewhere upstream.

job_ids = ["load_orders", "dedupe_users", "build_daily"]
minutes = [42, 7]
paired = {}
for job, mins in zip(job_ids, minutes):
    paired[job] = mins
    print("paired", job, mins)
print(len(job_ids), len(minutes), len(paired))
paired load_orders 42
paired dedupe_users 7
3 2 2

build_daily is never paired, and nothing in the output says so. In a pipeline this is silent data loss. When 2 lists are supposed to match, compare their lengths before zipping and stop the step if they differ:

def pair_runtimes(job_ids, minutes):
    if len(job_ids) != len(minutes):
        print("length mismatch:", len(job_ids), "jobs,", len(minutes), "runtimes")
        return None
    return dict(zip(job_ids, minutes))


print(pair_runtimes(["load_orders", "dedupe_users", "build_daily"], [42, 7]))

Output:

length mismatch: 3 jobs, 2 runtimes
None

Unzipping and sorting parallel lists

The * in a call spreads a list into separate arguments. zip(*pairs) therefore receives each pair as its own argument and groups all the first items together, then all the second items, splitting a list of pairs back into 2 tuples. Combined with sorted(), this orders 2 parallel lists by one of them while keeping every pairing intact.

job_ids = ["load_orders", "dedupe_users", "build_daily"]
minutes = [42, 7, 19]

pairs = sorted(zip(minutes, job_ids))
sorted_minutes, sorted_jobs = zip(*pairs)
print(pairs)
print(sorted_jobs)
print(sorted_minutes)

Output:

[(7, 'dedupe_users'), (19, 'build_daily'), (42, 'load_orders')]
('dedupe_users', 'build_daily', 'load_orders')
(7, 19, 42)

Putting minutes first in the zip() call makes each runtime the first item of its tuple, so plain sorted() orders by runtime with no key function, and the job name only matters on a tie. The results come back as tuples; pass them to list() if later code needs lists.

A column of counts is 1 value short. What does this program print?

names = ["orders", "users", "events"]
counts = [1200, 0]
pairs = zip(names, counts)
print(len(list(pairs)), len(list(pairs)))

Capstone: a ranking step end to end

The code quiz below puts all 5 ideas into one pipeline step, in the order a real job runs them: label the columns, convert the values, select the rows, rank the survivors. The debug drill after it returns to the tie rule from ranking on its own.

Ties deserve their own check because a wrong tie order raises nothing. Records built from parallel lists keep the order of those lists, so a track's position in the source decides which of 2 tied tracks takes first place under reverse=True. Flip the ties with [::-1] and the top of the report changes while every count stays the same.

About this lesson

This section is part of the Collections: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.