Parse Time vs Run Time

Concepts covered: Compilation

Every dbt command begins by reading the project. dbt opens each file under models/, macros/, seeds/, snapshots/ and tests/ and renders the Jinja in every model; each ref() and source() the render touches gets written down as an edge. That pass is the parse, and its product is the manifest, which records every node with its config and the nodes it depends on. The warehouse stays unreachable throughout, because dbt cannot schedule a single statement until the graph exists.

Execution starts once the manifest is complete. dbt opens a connection and, for each selected node in graph order, renders the model's Jinja a second time, now with the connection live, and sends the result as SQL. The 2 renders see a different value in one variable, execute: False during the parse, True during execution. Any Jinja that needs an answer from the warehouse (a list of values, a column list, a row count) has to test that variable before it asks.

The split explains 2 things people hit early. dbt ls on a project of 2,000 models returns in seconds without opening a connection, because listing needs only the manifest. And dbt compile is slower than its name suggests: it connects, because compiling means rendering with execute set to True, and every warehouse query in the project's Jinja fires. dbt parse never connects. It catches syntax errors in Jinja or YAML, and graph errors such as a ref() to a model that does not exist. It cannot see what fails only with execute on, such as a missing table or a column list a query returns, so in CI it is a necessary check and not a sufficient one.

2 renders of one file

{% if execute %}
  {% do log("rendering " ~ this ~ " with a live connection", info=True) %}
{% else %}
  {% do log("parsing " ~ this ~ " without one", info=True) %}
{% endif %}

select
    order_id,
    total_cents
from {{ ref('stg_orders') }}

dbt run on this model prints both messages: the parsing line first, while the manifest is being assembled, then the rendering line when the node's turn comes. dbt parse prints only the first. Both branches leave the SQL untouched, which is the pattern to copy: a branch on execute changes what Jinja computes, never what the SQL means. The second render is the one that reaches target/compiled/, so the file there always shows the True branch, and a failure that happens only offline never appears in it.

dbt saves the parse in target/partial_parse.msgpack and reparses only files that changed, so the parsing line shows on a first parse or after an edit to the model. dbt parse --no-partial-parse forces every file through again.

What each phase can see

flags.WHICH carries the command's name in both phases, so a macro can tell test from run and skip a grant during a test-only invocation. target.name works the same way and is the usual switch between dev and prod behaviour. Neither asks the warehouse anything, so neither needs execute. The macro below does need it, because it reads rows, and it shows where each line has to sit.

{% macro active_regions() %}
  {% set region_rel = ref('dim_region') %}
  {% set regions = [] %}
  {% if execute %}
    {% set result = run_query("select region from " ~ region_rel ~ " where is_active") %}
    {% set regions = result.columns[0].values() %}
  {% endif %}
  {{ return(regions) }}
{% endmacro %}

Line 2 runs in both phases and is what draws the edge from the calling model to dim_region. Moved inside the branch, it would run only under execution, after dbt had already fixed the build order, and the caller could build before dim_region had. Line 5 runs once, with a connection. The caller receives a list in both phases, so a for loop over the result renders cleanly offline and the parse completes. run_query itself is the subject of the next section.

About this lesson

This section is part of the Jinja and Macros: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.