Reading and Writing Files

Concepts covered: List Comprehension

Opening a file

open() takes a path and a mode and returns a file object: a handle with methods for reading and writing. The file's bytes stay on disk; the object remembers a position in them, which starts at 0. When the work is done, close() releases the handle, and anything still buffered in memory is written out.

f = open("plays.txt", "r")
text = f.read()
f.close()
print(text)

The runnable examples below use io.StringIO, which the io module provides: a file that lives in memory instead of on disk. Its read and readline methods work like a real file's, as do write and close, so every line transfers to a real open() call unchanged. The string handed to it is the file's content, and \n inside that string is the newline character, the point where a text editor would start a new line.

Reading moves a position

import io

f = io.StringIO("alpha\nbeta\ngamma\n")
print(repr(f.readline()))
print(repr(f.readline()))
print(repr(f.read()))
print(repr(f.read()))

Output:

'alpha\n'
'beta\n'
'gamma\n'
''

repr shows each string as Python would write it, so the \n ending each line is visible. The position advanced with every call: 2 readline calls consumed 2 lines, read() returned everything from the position to the end, and a second read() found nothing left and returned the empty string ''. A file never rewinds on its own.

One line at a time

read() loads the whole file into 1 string, so an export of 6 gigabytes costs 6 gigabytes of memory before any work starts. A file object can be looped over instead: for line in f: runs its indented block once per line, holding only the current line. for is a loop, which a later lesson covers in full; here it does 1 job, visiting each line in order from the first to the last.

import io

f = io.StringIO("alpha\nbeta\ngamma\n")
for line in f:
    print(line, end="")
f.close()
print(f.closed)

Output:

alpha
beta
gamma
True

Each line already ends with \n, and print adds a newline of its own, so end="" tells print to add nothing and the output keeps single spacing. After close(), the attribute f.closed reports True, and any further read on f raises ValueError: I/O operation on closed file.

Writing

Mode "w" empties the file the moment open() is called, before any write(): a program that opens counts.csv with "w" and crashes on the very next line has already destroyed the old counts. write() puts a string at the position and returns the number of characters written, which the interactive shell echoes as a bare number such as 5. It adds no newline, so each written line should end in \n, or the next write continues on the same line. getvalue() belongs to StringIO alone: it returns everything written, the content a disk file would hold after close().

import io

out = io.StringIO()
out.write("track_id,plays\n")
out.write("t1,3\n")
out.write("t2,7\n")
print(out.getvalue(), end="")

Output:

track_id,plays
t1,3
t2,7

Writing the result back safely

A job that opens its own output with "w" and crashes halfway leaves a half-written file, and the previous good version is already gone. The safe order is to write to a temporary name, close it, then rename it over the real name with os.replace. The rename is a single operation, so anything reading counts.csv sees either the old complete file or the new complete file, never a half of either.

import os

out = open("counts.tmp", "w")
out.write("track_id,plays\n")
out.write("t1,3\n")
out.close()
os.replace("counts.tmp", "counts.csv")

Opening a path that does not exist in mode "r" raises FileNotFoundError, a case for the previous section: a job on its first ever run can catch it and start from an empty result instead of dying.

try:
    f = open("counts.csv", "r")
    previous = f.read()
    f.close()
except FileNotFoundError:
    print("no previous counts, starting empty")
    previous = ""

Practice

What does this program print?

import io

f = io.StringIO("a\nb\n")
first = f.readline()
rest = f.read()
again = f.read()
print(repr(first), repr(rest), repr(again))

• 'a\n' 'a\nb\n' 'a\nb\n'
• 'a\n' 'b\n' 'b\n'
• 'a\n' 'b\n' ''
• 'a' 'b' ''

Double spacing is what a line-by-line read prints when end="" is left out. Each line arrives from the file with its own \n at the end, and print writes a second \n after it, so every line is followed by an empty one. Here is the 3-line file "alpha\nbeta\ngamma\n" traced through both calls:

About this lesson

This section is part of the Python Foundations: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.