Python Foundations: Advanced

The traceback lesson showed what happens when a line fails: Python prints the traceback and the program stops on that line. Every line after it never runs. For a script that parses 1 field typed at the keyboard, stopping is fine. For a nightly job on the 40,000th line of a 2 million line export, stopping throws away 39,999 lines of finished work because 1 line held 12x where a number belonged.

The last line of its traceback reads ValueError: invalid literal for int() with base 10: '12x'. ValueError is the exception's type, and the type is what an except clause matches on.

A try block holds the lines that might fail. An except block names an exception type and holds the lines to run when that type is raised inside the try. When the failing line raises, Python abandons the rest of the try block, runs the matching except block, and continues with the first line after it as though nothing had gone wrong.

print("parsed", count) never ran: the jump happened at int(raw), 1 line above it, and count was never assigned. Change raw to "12" and the output flips to parsed 12 followed by program continues, with the except block skipped entirely. The except block runs only when the try is abandoned, and the jump undoes nothing. Every line above the failing one has already run, and whatever those lines assigned or wrote stays done, which is why a try should hold only the lines that can fail. Below, the third line of the try fails after the first 2 have printed and assigned.

Writing except ValueError as err stores the exception object in err. Printing it gives the same message the traceback would have shown, so a log can record what went wrong while the program keeps going. Watch count in the run below: it keeps the 0 assigned before the try, because the assignment inside the try never completed.

An except clause catches only the type it names. except ValueError does nothing for a ZeroDivisionError, which then travels on as if no try were there and stops the program. Pick the type from the last line of the traceback you are trying to survive.

Several except clauses can follow 1 try, each for a different type; Python checks them top to bottom and runs the first that matches. Writing except: with no type catches everything, including the NameError from a typo, so the bug hides behind a skipped message for weeks while the output comes out wrong. Name the type.

A finally block after the except clauses runs whether the try succeeded, was rescued by an except, or raised something nothing caught. It is where cleanup goes: closing a file, releasing a connection, writing a final log line.

A handler can do more than print. When a field is missing or malformed, the except block can assign a fallback value, so the lines after the try always find the variable set. Below, an empty duration field becomes 0 seconds instead of stopping the job, and the program carries on with a usable number whichever block ran.

open() takes a path and a mode and returns a file object: a handle with methods for reading and writing. The file's bytes stay on disk; the object remembers a position in them, which starts at 0. When the work is done, close() releases the handle, and anything still buffered in memory is written out.

The runnable examples below use io.StringIO, which the io module provides: a file that lives in memory instead of on disk. Its read and readline methods work like a real file's, as do write and close, so every line transfers to a real open() call unchanged. The string handed to it is the file's content, and \n inside that string is the newline character, the point where a text editor would start a new line.

repr shows each string as Python would write it, so the \n ending each line is visible. The position advanced with every call: 2 readline calls consumed 2 lines, read() returned everything from the position to the end, and a second read() found nothing left and returned the empty string ''. A file never rewinds on its own.

read() loads the whole file into 1 string, so an export of 6 gigabytes costs 6 gigabytes of memory before any work starts. A file object can be looped over instead: for line in f: runs its indented block once per line, holding only the current line. for is a loop, which a later lesson covers in full; here it does 1 job, visiting each line in order from the first to the last.

Each line already ends with \n, and print adds a newline of its own, so end="" tells print to add nothing and the output keeps single spacing. After close(), the attribute f.closed reports True, and any further read on f raises ValueError: I/O operation on closed file.

Mode "w" empties the file the moment open() is called, before any write(): a program that opens counts.csv with "w" and crashes on the very next line has already destroyed the old counts. write() puts a string at the position and returns the number of characters written, which the interactive shell echoes as a bare number such as 5. It adds no newline, so each written line should end in \n, or the next write continues on the same line. getvalue() belongs to StringIO alone: it returns everything written, the content a disk file would hold after close().

A job that opens its own output with "w" and crashes halfway leaves a half-written file, and the previous good version is already gone. The safe order is to write to a temporary name, close it, then rename it over the real name with os.replace. The rename is a single operation, so anything reading counts.csv sees either the old complete file or the new complete file, never a half of either.

Opening a path that does not exist in mode "r" raises FileNotFoundError, a case for the previous section: a job on its first ever run can catch it and start from an empty result instead of dying.

Double spacing is what a line-by-line read prints when end="" is left out. Each line arrives from the file with its own \n at the end, and print writes a second \n after it, so every line is followed by an empty one. Here is the 3-line file "alpha\nbeta\ngamma\n" traced through both calls:

A play count belongs to a track id. Held in t1_name and t1_plays as separate variables, the 2 values have nothing in the program tying them together, and a job tracking 500 tracks needs 1,000 variables. A class is a blueprint for an object that holds a group of variables, called attributes, together with the functions that work on them, called methods. Building an object from the blueprint is called making an instance.

Line by line: class Tally: opens an indented block that defines the blueprint. def gives an indented block of code a name, and the block runs each time that name is called; functions get a lesson of their own later, and inside a class each one is a method. __init__ is the method Python calls when Tally("t1") runs, with self standing for the new object being built. self.name = name stores the argument on that object, and self.plays = 0 gives every new tally a starting count. The finished object is bound to t, and t.name reads the attribute back. type(t) reports the class it was built from, and that class's __name__ attribute holds its name, Tally, the way type(3).__name__ gives int.

a.add(3) is a shorthand: Python looks up add in Tally and calls it with a as self and 3 as count. The method changes a.plays and nothing else, so b keeps its own count. Two objects built from 1 class share the methods and hold separate attributes. The run below shows both objects on the stage as their counts move.

A method can hand a value back to its caller with return instead of printing it. The call then stands for that value, so it can be stored in a variable or passed straight to print. Below, share divides this tally's plays by a total it is given.

The self. prefix is what makes a change stick. Inside a method, ticks = ticks + 1 written without it names a temporary variable that has no value yet, so the call stops with UnboundLocalError: local variable 'ticks' referenced before assignment. Leaving self out of the header fails at the call instead, with the TypeError from the rule above. Both mistakes break the same contract: a method receives its object and writes to it.

f.read() returns the whole file as 1 string before any work starts, so the memory bill equals the file's size. for line in f pays for 1 line at a time. A generator is how you write a function with that second behavior: it produces values 1 at a time when asked and holds only what it needs to produce the next one.

A function that contains yield is a generator function. Calling it runs none of its lines; it returns a generator object. Each call to next() on that object runs the function until the next yield, hands out the value written there, and freezes with every variable intact until the following next().

created printed first, before anything inside the function ran. The first next(g) ran lines 2 and 3 and stopped at the yield, so print received the 1. The second next resumed at line 4. Line 6 has not run and will not until a third next arrives. A next after the final yield raises StopIteration to signal that the generator has nothing more. try and except handle it like any other exception.

A for loop calls next() for you and stops quietly at StopIteration, which makes a generator a natural fit for for. The generator below wraps a file: it reads 1 line and yields it converted, then reads the next line only when the loop asks. int() ignores surrounding whitespace, so the \n ending each line causes no error.

At no point does the program hold more than 1 line of the file. The run below stands a list of 3 lines in for the file and doubles each value on its way through. Watch n and total change: each time the generator yields, its frame steps off the stack with line and its place in the loop intact, then steps back on when the loop asks for the next value. Python discards that frame only after the last line has been read.

One yield written inside a for loop produces as many values as the loop has passes. Each request from the outer loop resumes the generator just after the yield, finishes that pass, takes the next line and pauses at the yield again. Here is plays(f) from earlier, 1 request at a time. Its input starts with 3 and 7 and ends with 5:

The file section called close() by hand. Between open() and close() sit the lines that do the work, and when 1 of them raises, Python jumps out of the block and close() never runs. The handle then stays open until Python gets around to discarding the file object, and nothing in the language promises when that will be. A job that opens a file per customer and keeps those objects alive, or runs on an interpreter that collects them late, leaks a handle on every error until the operating system refuses to open more. The finally block from the error section fixes this, and with is a shorter way to write the same guarantee.

with calls open() and binds the result to f. After the indented block runs, it closes f, even if the block raised. The 2 programs do identical work; the second cannot forget the cleanup, because the cleanup is part of the statement.

f is still a name after the block, and the object it points to has been closed; f.closed reports the change the moment the block ends.

with works on any object that defines 2 methods. __enter__ runs at the top of the block, and whatever it returns is what as binds. __exit__ runs when the block ends, receiving 3 arguments that describe the exception when 1 was raised and are all None otherwise. A file object defines both, and its __exit__ calls close(). A class of your own can define them for any resource that needs releasing.

The run below puts a division by zero inside the block and a try around the whole with. Watch the order: __exit__ prints before the handler outside does.

The division failed inside the block, so Python ran __exit__ first, passing the exception's class as exc_type, and only then let the ZeroDivisionError continue outward to the except wrapped around the with. Cleanup happened before the error was handled. An __exit__ that returns True tells Python the exception is dealt with, and nothing outside the with ever sees it; returning None, as this one does by having no return, lets the exception travel on.

A copy job needs a source and a destination open together. One with can hold both, separated by a comma, and closes them in reverse order when the block ends. An error on any line inside still closes both.

with calls __exit__ with 4 arguments every time: the object as self, then the 3 exception slots, filled with None when the block finished normally. A header that lists only self accepts 1 of them, so the call fails after the indented block has already run, with TypeError: __exit__() takes 1 positional argument but 4 were given. The traceback follows the block's own prints, and the cleanup line inside __exit__ never prints.