Full Refresh

Concepts covered: Incremental models

dbt run --full-refresh tells dbt to treat every incremental model in the run as though its table did not exist. is_incremental() returns false, the if block is dropped, and dbt drops the existing table and creates it again from the full SELECT, reading every source row. dbt build --full-refresh does the same and runs the tests after.

Situation	is_incremental()	Statement sent	plays afterward
Table absent, plain run	false	CREATE TABLE AS	every source row
Table present, plain run	true	INSERT INTO	old rows plus rows past the watermark
Table present, --full-refresh	false	drop, then CREATE TABLE AS	every source row

Reason 1: the SELECT changed

Spotify counts a stream once a track has played for 30 seconds, a rule it publishes for artists and royalties. Suppose plays has run for 30 nights keeping every event, and the team adds a WHERE on ms_played to apply the rule. A plain run applies the rule to tonight's rows and to nothing else: the 30 nights already in the table keep their short plays, because append never reads them again.

{{ config(materialized='incremental') }}

SELECT
    event_id,
    track_id,
    played_at,
    ms_played
FROM {{ source('app', 'stream_events') }}
WHERE ms_played >= 30000
{% if is_incremental() %}
  AND played_at > (SELECT MAX(played_at) FROM {{ this }})
{% endif %}

On the running example, say the rule lands between night 1 and night 2. Night 2's plain run judges events 4 and 5 under the rule and leaves events 1 to 3 as they were loaded:

event_id	ms_played	Loaded	After a plain run	After --full-refresh
1	214000	night 1, before the rule	kept	kept
2	9000	night 1, before the rule	kept, against the rule	dropped
3	198000	night 1, before the rule	kept	kept
4	45000	night 2, under the rule	inserted	kept
5	12000	night 2, under the rule	not inserted	not present

After the plain run the table holds 2 rules at once, and nobody reading it can tell which rows were loaded under which. One full refresh rebuilds every row under the SELECT as it now stands, drops event 2, and the table means 1 thing again. Any edit to the SELECT that changes what old rows would hold calls for the same treatment: a fixed calculation, a renamed value, a filter added or removed.

Reason 2: the source rewrote old rows

Event 2 was recorded with 9,000 ms played. The next day the app re-sends it with 186,000 ms, and the source team updates the row in place rather than adding one. The watermark filter looks only past 2026-03-01 21:05:00, and event 2 sits before it, so the row in plays keeps 9,000 under every plain run from now on. A full refresh reads the corrected value, because it reads everything.

A plain run has no way to find that row. Its filter looks only past the watermark, and dbt keeps no list of the source rows that changed, so the repair for a rewrite behind the line is to read everything again.

SELECT
    event_id,
    track_id,
    played_at,
    ms_played
FROM {{ source('app', 'stream_events') }}
{% if is_incremental() %}
WHERE played_at > (SELECT MAX(played_at) FROM {{ this }})
{% endif %}

dbt run
dbt run
dbt run --full-refresh

plays takes +materialized: incremental from dbt_project.yml. Before run 2 the source rewrites event 2 from 9000 to 186000 ms and gains no new plays, so the plain run inserts nothing and plays keeps 9000. Run 3 with --full-refresh reads every row and picks up 186000.

What a full refresh costs

A full refresh is night 1 again: every source row read and the whole table written. For the stream events source on night 2 it reads 90,300,000 rows against a normal night's 300,000, which is 301 times the rows read. The flag applies to every incremental model in the command, so a project with 12 incremental models pays 12 rebuilds in 1 run unless some of them refuse the flag.

Do

Refresh after any edit to the SELECT that changes what old rows would hold
Refresh when the source rewrote rows behind the watermark
Batch SELECT changes and pay for 1 rebuild instead of several
Run it through dbt build so the tests check the rebuilt table

Avoid

Expect a plain run to repair rows the table already holds
Leave 2 versions of a rule living in 1 table
Refresh a model whose source has dropped its oldest history
Pass --full-refresh on a schedule; each one is a decision someone makes

About this lesson

This section is part of the Incremental Models: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.