State Machine Patterns

Concepts covered: Classes

A batch run moves through a lifecycle. It waits in a queue and then runs. The run ends in success or failure, and a failed run may be retried. Code that tracks this with separate booleans can represent combinations no real run can be in. The flags here are is_running, is_done and has_failed, and a single forgotten reset is enough to produce one.

• 8
• 4

Take a failure handler that sets has_failed = True and forgets is_running = False. A legal run sets at most 1 flag. Queued leaves all 3 False; running sets only is_running. Succeeded and failed each set only their own flag, is_done and has_failed respectively. This handler leaves is_running and has_failed both True, a combination no run can be in. Every report that checks is_running now counts a dead run as live, and no single variable answers the question of where the run is.

A single variable, a fixed set of states

A state machine replaces the flags with a single variable. Its value is always 1 of 4 names: queued, running, succeeded or failed. Events come from outside the machine, and start and fail are 2 of them. A transition table says which event moves which state where. Any pair missing from the table is rejected, so no sequence of events can reach an impossible state.

The class below holds the state and exposes handle as the only way to change it. Every branch in its if-elif chain pairs a current state with an event, and the final else catches each combination the table leaves out.

__init__ runs once, when PipelineRun() creates a run, and sets its starting state. self.state is the variable each run carries with it, so 2 runs can sit in different states. handle is a function attached to the run: run.handle("start") calls it with event set to "start", and self is that same run.

class PipelineRun:
    def __init__(self):
        self.state = "queued"

    def handle(self, event):
        old = self.state
        if self.state == "queued" and event == "start":
            self.state = "running"
        elif self.state == "running" and event == "succeed":
            self.state = "succeeded"
        elif self.state == "running" and event == "fail":
            self.state = "failed"
        elif self.state == "failed" and event == "retry":
            self.state = "queued"
        else:
            print("rejected:", event, "in", old)
        if self.state != old:
            print(old, "->", self.state)


run = PipelineRun()
run.handle("start")
run.handle("start")
run.handle("fail")
run.handle("retry")
run.handle("start")
run.handle("succeed")
run.handle("retry")

queued -> running
rejected: start in running
running -> failed
failed -> queued
queued -> running
running -> succeeded
rejected: retry in succeeded

Watch the second and last events. start arrives while the run is already running, and retry arrives after it succeeded. Neither pair has a branch, so both are refused and self.state keeps its value. With flags, the same mistakes would have flipped a boolean and left the run in a combination no report expects.

A fresh PipelineRun from the animation receives start, then fail, then succeed. What is run.state afterwards?

• succeeded, because succeed is an event this machine accepts
• failed, because no branch pairs the failed state with succeed
• queued, because a failed run goes back to the queue on any event
• running, because succeed is rejected and start left it there

Extending the machine takes one new branch per legal pair. The branch tests both the current state and the event with and, assigns the next state, and goes above the final else, so every pair it doesn't name is still refused.

About this lesson

This section is part of the Control Flow: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.