String Encoding

Concepts covered: Type Conversion

Computers store text as bytes, and encoding is the rule that converts characters into them. Once you know the encoding of whatever you're reading, a whole class of confusing bugs goes away. That holds for files and API responses as much as for database columns.

Strings vs bytes

Python has 2 types for text-like data: str (Unicode strings) and bytes (raw byte sequences). They aren't interchangeable.

text = "Hello"
data = b"Hello"

print(type(text))
print(type(data))

Output:

<class 'str'>
<class 'bytes'>

The b prefix creates a bytes object. Strings are for text; bytes are for binary data like files, network data, or images.

encode() and decode()

.encode() converts a string to bytes. .decode() converts bytes back to a string.

text = "Hello, World!"

# String to bytes
encoded = text.encode("utf-8")
print(encoded)
print(type(encoded))

# Bytes back to string
decoded = encoded.decode("utf-8")
print(decoded)

Output:

b'Hello, World!'
<class 'bytes'>
Hello, World!

UTF-8 is the most common encoding. It handles all Unicode characters and is the default for web and most modern systems.

Unicode characters

Unicode supports characters from all languages and emojis:

greeting = "Hello"
encoded = greeting.encode("utf-8")

print(f"String: {greeting}")
print(f"Length: {len(greeting)}")
print(f"Bytes:  {encoded}")
print(f"Byte length: {len(encoded)}")

Output:

String: Hello
Length: 5
Bytes:  b'Hello'
Byte length: 5

For ASCII characters, the string length and byte length are the same. But for international characters, len() counts characters while the encoded byte length can be 2-4x larger. This difference matters when you work with file sizes, network buffers, or database column limits.

Common encodings

• UTF-8 - Most common, handles all Unicode, variable length
• ASCII - English only, 1 byte per character, oldest encoding
• Latin-1 - Western European, 1 byte, also called ISO-8859-1
• UTF-16 - Fixed 2+ bytes, used by Windows internally

Using the wrong encoding produces a distinctive class of bugs.

Handling encoding errors

Sometimes bytes contain invalid sequences for an encoding. Handle errors with the errors parameter:

bad_bytes = b"\xff\xfe"

text = bad_bytes.decode("utf-8", errors="replace")
print(repr(text))

text = bad_bytes.decode("utf-8", errors="ignore")
print(repr(text))

Output:

'��'
''

errors="replace" substitutes invalid bytes with a replacement character. errors="ignore" skips invalid bytes entirely.

The reference below shows when to reach for each strategy.

Choosing an error strategy

• strict (default) - Raise an error; best for data you control
• replace - Insert a marker character; good for display or logging
• ignore - Silently drop bad bytes; use with caution
• backslashreplace - Escape invalid bytes; great for debugging

When string methods aren't enough

For complex text patterns, Python provides the re (regular expression) module. While full regex is beyond this lesson, here's a preview of what's possible:

import re

text = "Contact us at support@example.com or sales@example.com"

emails = re.findall(r'\b\w+@\w+\.\w+\b', text)
print(emails)

cleaned = re.sub(r'\s+', ' ', "too   many    spaces")
print(cleaned)

Output:

['support@example.com', 'sales@example.com']
too many spaces

A regular expression describes the shape of a string. Email addresses have a predictable shape, and so do phone numbers and URLs, so a regex can pull any of them out of unstructured text.

String Methods

• Simple, readable code
• Fast for basic operations
• Use for: split, join, replace

Regular Expressions

• Complex pattern matching
• Steeper learning curve
• Use for: validation, extraction

The scenario below combines these techniques in a data engineering task.

Real data engineering means making smart choices about text processing. Walk through this scenario.

Choosing the right string processing tool, whether split, slicing, or regex, depends on the structure of your data. Delimiter-based data is easiest to handle with split(). Fixed-width data suits slicing. Variable-structure or pattern-based data calls for regular expressions.

Any text parser needs a test suite, because log formats change, new systems start sending data, and production data turns up edge cases nobody planned for. Without tests, a regression can corrupt your downstream analysis silently.

About this lesson

This section is part of the Strings: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.