Searching in Strings

Concepts covered: String Methods

Slicing needs positions you already know, and real data rarely hands them over: a log level lands wherever the timestamp ends, and an email's @ can be anywhere. Searching answers 2 questions: is this text present, and where does it begin. The in operator answers the first with True or False.

line = "2024-03-15 08:42:10 ERROR disk full on node-7"
print("ERROR" in line)
print("error" in line)
print("error" in line.lower())
print("node-9" not in line)

Output:

True
False
True
True

Every search is case-sensitive, so "error" misses ERROR. Lowering the line first, with the lower() method you already know, makes the match ignore case. not in is the negated form, and it reads closer to English than not ("node-9" in line).

The empty string counts as present everywhere: "" in line is True for every line, even an empty one. A search term read from a blank setting therefore matches everything, so check len(term) > 0 before searching.

Finding the position with find and index

s.find(sub) returns the index where the first match of sub begins, or -1 when there is none. s.rfind(sub) searches from the right and returns where the last match begins. s.index(sub) gives the same answer as find while the text is present and raises ValueError when it's missing. A Spotify track URI, spotify:track:6rqhFgbbKwnb9MLmUQDhG6, holds 2 colons to search for.

uri = "spotify:track:6rqhFgbbKwnb9MLmUQDhG6"
print(uri.find(":"))
print(uri.rfind(":"))
print(uri.find("track"))
print(uri.find("album"))
print(uri.index("album"))

The first colon is at 7 and the last at 13, so the ID starts at 14. find("track") returns 8, the position of the t that opens the match. Pick by what a missing match means. Use find and test for -1 when absence is normal, as in a log line with no level. When absence means the record is broken, index halts the program at that line, which beats writing a wrong value downstream.

Slicing between 2 search results

A pair of searches marks 2 boundaries, and a slice reads what lies between them. The text between a URI's colons names what it points to, and the text after a filename's last dot is its extension:

uri = "spotify:album:4aawyAB9vmqN3uQ7FjRGTy"
first = uri.find(":")
last = uri.rfind(":")
print(first, last)
print(uri[first + 1:last])
name = "orders.2024-03-15.csv"
print(name[name.find(".") + 1:])
print(name[name.rfind(".") + 1:])

Output:

7 13
album
2024-03-15.csv
csv

first + 1 steps past the first colon so it stays out of the result, and last as the stop keeps the second colon out because the stop is excluded. The code never mentions the word's length, so a playlist URI yields playlist with the same 2 searches. The filename shows why the direction matters: find stops at the first dot and returns 2024-03-15.csv, while rfind lands on the last dot and returns csv.

The -1 trap

def after_colon(s):
    return s[s.find(":") + 1:]

print(after_colon("user:42"))
print(after_colon("user42"))

Output:

42
user42

For "user42", find returns -1, and -1 + 1 is 0, so the slice becomes s[0:], the whole input, returned as if it were the answer with nothing to flag it. Used as an index, the same -1 reads the last character, because -1 is a valid negative index. Test the result before treating it as a position:

def after_colon(s):
    pos = s.find(":")
    if pos == -1:
        return None
    return s[pos + 1:]

Searching again from a position

find takes an optional start index: s.find(sub, start) ignores everything before start. Starting each search 1 position after the previous hit walks through every match, and the loop ends when find returns -1.

log = "ERR ok ERR ok ok ERR"
hits = 0
pos = log.find("ERR")
while pos != -1:
    print(pos)
    hits += 1
    pos = log.find("ERR", pos + 1)
print(hits, log.count("ERR"))

0
7
17
3 3

The loop counts 3 matches, the last starting at 17 after ones at 0 and 7, which agrees with log.count("ERR"). count reports matches without overlap: "aaaa".count("aa") is 2, because the scan resumes after the end of each match rather than 1 character later.

Checking the start and the end

path = "raw/orders/2024-03-15/part-0001.csv"
print(path.startswith("raw/"))
print(path.endswith(".csv"))
print(path.endswith(".CSV"))
print(path.startswith("orders", 4))
print(path[4:10] == "orders")

Output:

True
True
False
True
True

startswith and endswith test a prefix or suffix with no length arithmetic. path[-4:] == ".csv" gives the same answer, but its 4 must be edited by hand when the suffix becomes .json, while endswith(".json") carries its own length. Both accept a start index, so startswith("orders", 4) checks the text that begins at index 4.

What does this program print?

s = "banana"
print(s.find("an"), s.rfind("an"), s.count("ana"), s.find("an", 2))

• 1 3 2 3
• 1 4 1 3
• 1 3 1 1
• 1 3 1 3

About this lesson

This section is part of the Strings: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.