The Recompute Cost

Concepts covered: Spark Execution Model

Each check replays every earlier round

The stopping check ends in first(), an action, and an action runs a plan from its sources. At round k, the check's plan begins at the nodes scan and holds k joins and k groupBys, so Spark recomputes rounds 1 through k before adding up a single label. The job for round k-1 sent its sum to the driver and kept nothing a later job could start from.

Put a price on it. Suppose a round of join and groupBy over the member graph takes 2 minutes of cluster time. The check at round k then costs about 2k minutes, and the run pays for every check along the way:

Round	Rounds its check replays	Check time	Rounds of work so far
1	1	2 min	1
2	2	4 min	3
10	10	20 min	55
20	20	40 min	210
30	30	60 min	465

After n checks the total is n(n+1)/2 rounds, which grows with the square of n. 30 rounds of progress cost 465 rounds of work, or 930 minutes at 2 minutes a round, where computing each round once would take 60 minutes. Doubling the run to 60 rounds roughly quadruples the bill, to 1,830 rounds.

15.5x
29

Narrow loops pay less of this bill on executors. A loop of withColumn rewrites like the one in the previous section's cell replays cheaply, because pipelining fuses its steps into a single pass over each row. Its cost lands on the driver, which still plans every step. Loops whose rounds hold joins and groupBys pay on both sides.

Fewer actions, same plan

Running fewer actions cuts the cost without any new API. Here are 3 loop shapes over 30 rounds, priced in rounds of work.

Loop shape	Rounds of work	Joins in the last plan
Check after every round	465	30
Check after every 5th round	105	30
30 fixed rounds, one action at the end	30	30

Checking less often cuts executor work, but the final action still plans and runs a 30-join chain, and a fixed round count either stops too early or wastes rounds after the labels settle. Only a checkpoint bounds the depth of the plan itself.

The bill shows up in 3 places

Where repeated rounds cost you

Executors: Each check rescans edges and reruns every earlier join and groupBy. Round k's job reads the full edges table k times.
Driver: Planning round k's check means working through k joins. Gaps between jobs widen while executors sit idle.
Recovery: If an executor dies mid-job, the DAGScheduler resubmits the parent stages whose output it held. In a deep plan, those parents reach back through earlier rounds.

Reading it in the Spark UI

Where to look	Loop with its plan depth capped	Loop replaying every round
Jobs tab, check durations	Rise for a few rounds, then drop back to one round's time	A straight climb: 2, 4, 6 ... 60 minutes
Stages per check job	Stays under a fixed cap	More each round, a join and groupBy's worth
Stages tab, edges input	Read a capped number of times per check	Read k times by round k's check
Event timeline	Short, steady gaps	Widening gaps with idle executors

A climbing line on the Jobs tab while input size holds still points to lineage before it points to data. To confirm, open 2 check jobs a few rounds apart and compare their DAG visualizations. If the later one holds more joins, the plan is carrying history from round to round.

dims holds 8 DataFrames, and each join takes about 1 minute when run once. Roughly how many join-minutes does this loop spend on executors?

events = spark.table("user_events")
for i in range(8):
    events = events.join(dims[i], "user_id")
    print(i, events.count())

8 join-minutes, since each count() runs only its own round's join
16 join-minutes, since each count() reruns its own join and the one before
64 join-minutes, since each count() reruns all 8 joins of the loop
36 join-minutes, since the count() at round k reruns all k joins so far

About this lesson

This section is part of the Lazy Evaluation: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.