Local vs Reliable Checkpoints
Concepts covered: Caching & Persistence
Executor storage or shared storage
df.localCheckpoint() cuts lineage the same way checkpoint() does: the DataFrame it returns has a plan that starts from saved rows. What differs is where those rows live. A local save keeps each partition in the storage of the executor that computed it, in memory with spill to that executor's local disk. A reliable save writes files to the checkpoint directory on shared storage.
| Property | localCheckpoint() | checkpoint() |
|---|---|---|
| Where rows live | Executor memory and local disk | Files in the checkpoint directory |
| Needs `setCheckpointDir` | No | Yes |
| Passes over the plan per save | 1 | 2, unless the rows are already in executor storage |
| Write target | Executor memory or local disk | Shared storage, usually slower |
| After an executor is lost | Its partitions are gone | Any executor reads the files |
| Cleanup | Freed after the driver drops its last reference to it | Left on storage unless `spark.cleaner.referenceTracking.cleanCheckpoints` is `true` |
Reliable saves outlive the run. With spark.cleaner.referenceTracking.cleanCheckpoints at its default of false, every save's files stay in the checkpoint directory after the job ends, so a nightly job that saves 6 times a night leaves 180 sets of files behind after a month. Give each run its own subdirectory and delete it when the run succeeds.
Losing an executor after a local save
Normally a lost executor costs only time: the DAGScheduler notices the missing output and reruns the stages that produced it, using their lineage. A local save removed that lineage on purpose. If the executor holding partition 2 of the round-25 labels dies, no plan exists that can rebuild partition 2, so the task that needs it fails with an error saying the checkpoint block was not found because the executor that saved it is no longer alive.
Retries cannot fix this, because every attempt looks for the same missing block. After spark.task.maxFailures attempts, 4 by default, the stage is aborted and the job fails in round 26 with 25 rounds of progress lost. Recovery means rerunning from round 1.
A 20-round loop runs labels = new_labels.localCheckpoint() every 5 rounds. During round 14, the executor holding partition 6 of the round-10 save dies. What happens?
- Spark rebuilds partition 6 from round 5's save, then reruns rounds 6 to 10 for it
- The task retries on another executor and reads a replica of partition 6
- Spark reruns rounds 1 to 10 for partition 6 from the original tables
- The job fails, since partition 6 has no copy and no lineage to rebuild it from
Choosing between local and reliable
Back to the 30-round job with a save every 5 rounds. A local save stores rows as the counting pass computes them, so it replays its 5 rounds once. Each block then costs 10 rounds of checks plus 5 for the save: 90 rounds over the run against 120 with reliable saves, and no writes to shared storage. The price is exposure. With local saves, a lost executor fails the job whenever it held partitions of the latest save, which most executors do; with reliable saves, a lost executor costs at most the rounds since the last save.
- 3 hours
- 25%
A risk team's job runs 40 rounds of fraud-ring detection for 5 hours on 200 executors, and 3 of last month's runs lost an executor partway through. Which save strategy fits?
- localCheckpoint() every 5 rounds, because it skips the second pass a reliable save makes
- No saves at all, because retries rebuild any lost partition from its lineage
- checkpoint() every 5 rounds, because a lost executor then costs only rounds since the last save
- localCheckpoint() every round, because frequent saves limit what a lost executor can take
About this lesson
This section is part of the Lazy Evaluation: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.