DAG vs Lineage

Concepts covered: Spark Execution Model

Two names come up for the graph behind a DataFrame, and they refer to different objects with different lifetimes. Confusing them leads to wrong guesses about when work happens and what Spark repeats after a failure.

Lineage: the recipe a DataFrame carries

Lineage is the chain of parent links from the first section: each DataFrame records its operation and its parents, all the way back to the source. It exists from the line that defines the DataFrame and lasts as long as the DataFrame does. It holds operations and parent links, with no stages or tasks. At the action, Spark expands it into datasets with partition-level dependencies: a narrow link says output partition 3 needs input partition 3, and a shuffle link says it needs a piece of every input partition.

The DAG: the schedule for 1 job

The scheduler builds a DAG when an action submits a job. It takes the part of the lineage that action needs, cuts it at shuffle links, and produces stages and the tasks inside them. The DAG belongs to that job and is gone when the job ends. Two actions on the same DataFrame produce 2 jobs and 2 DAGs from 1 unchanged lineage, which is the re-execution from the beginner lessons seen from the scheduler's side.

Lineage

• Recorded as each transformation is defined
• Lasts as long as the DataFrame
• Operations and the parents they read
• Answers: how is this partition rebuilt?
• Visible in your code as the chain of variables

DAG

• Built when an action submits a job
• Discarded when that job ends
• Stages and the tasks inside them
• Answers: what runs, in what order?
• Visible in the Spark UI, 1 diagram per job

Following both through 4 lines

clean = page_views.filter(F.col("dur_ms") > 0)
per_page = clean.groupBy("page_url").agg(F.count(F.lit(1)).alias("views"))
per_page.collect()
per_page.filter(F.col("views") > 100).collect()

Job 2 repeats the read, the filter and the count, because nothing from job 1 was kept. Reading the table this way shows where the cost sits: lineage grows with every line and costs almost nothing to extend, while each DAG is a full run of everything upstream of its action.

What each one does after a lost executor

When an executor dies, the partitions it held are gone, and Spark rebuilds them from lineage. A partition behind narrow links is rebuilt by re-running its chain from the 1 source partition it came from, and nothing else. A partition behind a shuffle link needs the shuffle files the parent stage wrote; if those lived on the lost executor too, the scheduler re-runs the parent-stage tasks that produced them.

So lineage says what to recompute, and the DAG's stage boundaries decide how far back the recomputation reaches. A loss inside a pipeline of narrow steps costs a few task re-runs. A loss after 3 shuffles can pull work back through all 3 of them.

These 3 lines have run and no action has been called. What does Spark hold for top?

clean = page_views.filter(F.col("dur_ms") > 0)
per_page = clean.groupBy("page_url").agg(F.count(F.lit(1)).alias("views"))
top = per_page.filter(F.col("views") > 100)

• A DAG of 2 stages, built when per_page was defined, waiting for an action to run it
• Nothing yet: Spark records the chain of operations only when the first action begins
• A lineage of 3 operations back to page_views, and no DAG until an action asks for top
• A lineage and a DAG, both built line by line, so the action only has to launch the tasks

About this lesson

This section is part of the Lazy Evaluation: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.