Combining Lists
Concepts covered: List Modification
A pipeline rarely receives everything in a single list. Records arrive in batches, a list per file or per day, and a later step needs them as one sequence. Python has several ways to put lists together, and the difference behind many bugs is where the result ends up: in a new list, or in an existing list that grows?
The + operator builds a new list
monday = ["r101", "r102", "r103"]
tuesday = ["r201", "r202"]
week = monday + tuesday
print(week)
print(monday)
print(tuesday)
print(tuesday + monday)Output:
['r101', 'r102', 'r103', 'r201', 'r202'] ['r101', 'r102', 'r103'] ['r201', 'r202'] ['r201', 'r202', 'r101', 'r102', 'r103']
+ copies the items of the left list, then the items of the right list, into a brand-new list. Neither input changes, and order follows the operands: tuesday + monday puts Tuesday first. Both sides must be lists. monday + "r104" raises TypeError: can only concatenate list (not "str") to list, so wrap a single item in brackets: monday + ["r104"].
extend() grows the list you already have
team_a = ["Alice", "Bob", "Charlie"]
team_b = ["Diana", "Eve"]
print("Team A before:", team_a)
print("Team B before:", team_b)
team_a.extend(team_b)
print("Team A after:", team_a)
print("Team B after:", team_b)Team A before: ['Alice', 'Bob', 'Charlie'] Team B before: ['Diana', 'Eve'] Team A after: ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'] Team B after: ['Diana', 'Eve']
team_a.extend(team_b) adds each item of team_b to the end of team_a. Afterwards team_a holds 5 names and team_b still holds its 2, and no third list exists. For lists, team_a += team_b behaves like extend(): it adds to the existing list rather than building a new one.
append() vs extend()
batch = ["r101", "r102"]
late = ["r103", "r104"]
batch.append(late)
print(batch)
print(len(batch))
batch = ["r101", "r102"]
batch.extend(late)
print(batch)
print(len(batch))Output:
['r101', 'r102', ['r103', 'r104']] 3 ['r101', 'r102', 'r103', 'r104'] 4
append() adds its argument as a single item, whatever that argument is. Given a list, it adds the whole list as one element, so batch ends with a nested list and has a length of 3. A loop over that batch would hand its third pass a list where it expects an ID. extend() walks through its argument and adds each item separately, giving the 4 IDs you meant.
Repeating a list with *
print([0] * 4)
print(["a", "b"] * 3)
print([] * 10)Output:
[0, 0, 0, 0] ['a', 'b', 'a', 'b', 'a', 'b'] []
Multiplying a list by an integer n builds a new list holding its items n times over. [0] * 7 is a common way to start 7 daily counters before a loop fills them in. Numbers and strings repeat this way, and so does None. An empty list repeated any number of times stays empty.
| Expression | Result | Changes an existing list? |
|---|---|---|
| `a + b` | New list: items of a, then b | No |
| `a += b` | a grows by the items of b | Yes, a |
| `a.extend(b)` | a grows by the items of b | Yes, a |
| `a.append(b)` | a grows by 1 item: the list b | Yes, a |
| `a * 3` | New list: items of a, 3 times | No |
Gathering many lists in a loop
batches = [["r101", "r102"], ["r201"], [], ["r301", "r302"]]
all_ids = []
for batch in batches:
all_ids.extend(batch)
print(all_ids)
print(len(all_ids))Output:
['r101', 'r102', 'r201', 'r301', 'r302'] 5
Starting from an empty list and extending once per batch is the standard pattern for gathering many lists into one. The empty batch adds nothing and raises no error. Writing all_ids = all_ids + batch gives the same result, but builds a fresh list on every pass and copies everything gathered so far, which slows down as the total grows.
Which list does each line touch?
def with_header(rows):
return ["id"] + rows
data = ["r101", "r102"]
out = with_header(data)
print(out)
print(data)
data.extend(["r103"])
print(out)
print(data)Output:
['id', 'r101', 'r102'] ['r101', 'r102'] ['id', 'r101', 'r102'] ['r101', 'r102', 'r103']
When several names are in play, track each one line by line. with_header() combines with +, so it hands back a separate list and leaves its argument with the 2 IDs it started with. Extending data afterwards changes data alone: out still prints 3 items. A function that builds its result with + is the safer choice whenever the caller still needs its input as it was.
What does this program print?
a = [1, 2]
b = a + [3]
a.extend([4])
print(a, b)- [1, 2, 4] [1, 2, 3]
- [1, 2, 4] [1, 2, 3, 4]
- [1, 2, 3, 4] [1, 2, 3]
- [1, 2, [4]] [1, 2, 3]
About this lesson
This section is part of the Lists: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.