List Comprehensions

Concepts covered: List Comprehension

A list comprehension builds a new list from an existing iterable in a single expression: [expression for item in iterable]. Python runs the loop, evaluates the expression once per item, and collects each result into a fresh list, in source order. Code that used to start with an empty list and fill it in a loop with append becomes a line that states what the list contains.

amounts_cents = [1999, 450, 12000, 75]

dollars = []
for cents in amounts_cents:
    dollars.append(cents / 100)

dollars_again = [cents / 100 for cents in amounts_cents]

print(dollars)
print(dollars_again)
print(dollars == dollars_again)

Output:

[19.99, 4.5, 120.0, 0.75]
[19.99, 4.5, 120.0, 0.75]
True

Both lists are equal, and amounts_cents is untouched: the comprehension only reads its source. The table names the parts you will find in every comprehension.

Filtering with a trailing if

An if clause after the for decides which items reach the expression. Items that fail it are skipped, so the result can be shorter than the source. This payments feed mixes charges with refunds, stored as negative cents, and zero-value test events. The report wants charges only, in dollars.

amounts_cents = [1999, -450, 12000, 0, -75, 300]

charges = [c / 100 for c in amounts_cents if c > 0]
print(charges)
print(len(amounts_cents), len(charges))

Output:

[19.99, 120.0, 3.0]
6 3

Transforming with if-else in front

To keep every item but compute it differently, put a conditional expression in the expression slot: a if condition else b. It's the conditional assignment you already write on a single line, placed before the for. Every source item produces an output, so the length never changes.

amounts_cents = [1999, -450, 12000, 0]

labels = ["refund" if c < 0 else "charge" for c in amounts_cents]
print(labels)
print(len(labels))

Output:

['charge', 'refund', 'charge', 'charge']
4

if after for: filter

• [c for c in xs if c > 0]
• Decides whether an item appears
• Takes no else; adding one is a SyntaxError
• Result can be shorter than the source

if-else before for: transform

• [c if c > 0 else 0 for c in xs]
• Decides what an item becomes
• The else is required
• Result always matches the source's length

Both can appear together: ["big" if c > 10000 else "small" for c in xs if c > 0] drops non-positive amounts, then labels the rest.

Several for clauses

A comprehension can hold more than one for. The clauses run in the order you would nest the loops, outer loop first, so reading them left to right as indented loops tells you what comes out. A common use turns a list of batches into a list of rows.

batches = [["a1", "a2"], ["b1"], ["c1", "c2", "c3"]]

rows = [row for batch in batches for row in batch]
print(rows)

rows_loop = []
for batch in batches:
    for row in batch:
        rows_loop.append(row)
print(rows == rows_loop)

Output:

['a1', 'a2', 'b1', 'c1', 'c2', 'c3']
True

Reverse the clauses to [row for row in batch for batch in batches] and Python evaluates batch before any clause defines it: a NameError, or a silent wrong answer if an earlier batch variable is still around. A comprehension can also sit in the expression slot of another. [[r * c for c in range(3)] for r in range(2)] gives [[0, 0, 0], [0, 1, 2]], and every pass of the outer loop builds a separate inner list, a detail the next section depends on.

The loop variable stays inside

A for loop leaves its variable bound after it ends. A comprehension runs in its own scope, so its loop variable vanishes with it and never overwrites a name outside.

r = "outside"
doubled = [r * 2 for r in [1, 2, 3]]
print(doubled)
print(r)

for r in [1, 2, 3]:
    pass
print(r)

Output:

[2, 4, 6]
outside
3

When a loop reads better

A comprehension exists to produce a list. [print(r) for r in rows] prints each row and also builds a throwaway list of None values, because print returns None. Write a for loop when the point is a side effect, or when the logic needs several conditions and transforms that would crowd a single line. When you only need to scan the values once, parentheses instead of brackets give a generator expression, as in any(c < 0 for c in amounts_cents), which stops at the first refund and builds no list.

Do

• Use a comprehension when the goal is a new list
• Filter with a trailing if; transform with if-else up front
• Order for clauses the way you would nest the loops

Avoid

• Don't call functions for their side effects inside a comprehension
• Don't put else after a trailing filter if
• Don't expect a comprehension's loop variable to exist afterwards

What does this program print?

scores = [90, 45, 72, 30, 88]
result = ["pass" if s >= 50 else "fail" for s in scores if s > 40]
print(result)

• ['pass', 'fail', 'pass', 'fail', 'pass']
• ['pass', 'fail', 'pass', 'pass']
• ['pass', 'pass', 'pass']
• SyntaxError: only one if is allowed

When you write a comprehension that both filters and converts, fill its slots by asking 2 questions in order: which items should survive, and what each survivor should become. The first answer is the condition after the for, and the second is the expression before it. Only survivors reach the expression, so a division placed there never runs on an item the filter dropped.

About this lesson

This section is part of the Lists: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.