Warn or Error

Concepts covered: Test severity and blocking

A test's severity decides what dbt does once the test's query has returned rows. The default severity is error: dbt prints FAIL with the row count, marks the test failed, skips every node downstream of the model the test guards, and ends dbt build with a nonzero exit code, so the scheduler records a failed run. The other severity is warn: dbt prints WARN with the same row count, builds every downstream node as if the test had passed, counts the test under warnings in the summary line, and exits with code 0. The query is identical under both; only what happens to its result changes.

Severity is a config on the test, so in a properties file it sits under config: beside the test it changes. Here the key tests on fct_order_lines keep the default, and the accepted_values test on shipping_region becomes a warning:

models:
  - name: fct_order_lines
    columns:
      - name: order_line_key
        data_tests:
          - unique
          - not_null
      - name: shipping_region
        data_tests:
          - accepted_values:
              arguments:
                values: ['NA', 'EU', 'APAC', 'LATAM']
              config:
                severity: warn

The log tells the 2 cases apart on the test's own line and again in the summary. An error reads FAIL 3 not_null_fct_order_lines_merchant_id and is followed by SKIP lines for every model behind it; a warning reads WARN 3 accepted_values_fct_order_lines_shipping_region and the models behind it show OK as usual. Under dbt test there are no models to skip, so the difference is the exit code alone, and a scheduler that triggers downstream jobs on exit code 0 will trigger them after a warning.

Which severity a rule deserves

The question that picks a severity is what a downstream reader would see if the build continued. A null merchant_id on fct_order_lines means fct_payouts sums that line into no merchant's payout, so money leaves a real total; that test is an error, because a build that continues produces a payout table someone acts on. A shipping_region outside the 4 listed values means 1 row lands in an other bucket on a dashboard; every other row on that dashboard is still right and nobody is paid from it, so that test is a warning that tells the team a new region appeared.

Most tests fall into 1 of 3 cases. If a failure would make a downstream number wrong, or feed a wrong row into a system outside the warehouse, the test is an error. A condition in the source data that the team has decided to tolerate while it is fixed upstream gets a warning. When nobody would act on the failure either way, delete the test, which section 4 returns to.

A default for a whole folder

Severity can also be set once in dbt_project.yml, scoped by folder, and a test's own config: overrides the folder's value. A project whose staging tests describe the quality of raw feeds, while its marts carry guarantees, writes:

data_tests:
  shop:
    staging:
      +severity: warn
    marts:
      +severity: error

The folder default replaces severity: warn written on 60 staging tests with 1 line, and it makes the policy readable in one place. It also makes an override stand out: a staging test that carries severity: error in its own config: is announcing that this one rule is a guarantee, and a comment saying why belongs next to it.

The animation below builds stg_orders and fct_orders from the same 5 raw rows, 1 of them with a null customer_id, under both severities. Switch between the versions and watch what happens to fct_orders once not_null has found that row.

{{ config(materialized='table') }}

select
    order_id,
    customer_id,
    order_total,
    order_total * 0.029 as processing_fee
from {{ ref('stg_orders') }}

Lowering a severity is a decision

Changing a test from error to warn is the quickest way to turn a red build green, and the change outlives the morning it was made. A test lowered to warn during an incident keeps warning after the incident, and 3 months later nobody remembers that it once stopped the build. Record the reason in the properties file beside the severity line, with a date, so the next engineer can decide whether the reason still holds.

severity: error

• Prints FAIL with the row count
• Skips every model downstream of the tested model
• Exit code 1, the run is reported failed
• For rules a downstream number depends on

severity: warn

• Prints WARN with the row count
• Builds downstream models as if the test passed
• Exit code 0, the run is reported successful
• For known conditions someone is tracking

About this lesson

This section is part of the Testing: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.