Data Observability Tools & Data Pipeline Monitoring Tools (2026)

Job monitoring is the layer most teams already have. Airflow emits metrics through StatsD or OpenTelemetry: ti_failures counts task failures by DAG and task, dagrun.duration.success.{dag_id} times each run, and scheduler_heartbeat shows whether the scheduler is alive. You graph and alert on them in whatever metrics stack you run.

Dagster moves one step toward the data. An asset check, written with @asset_check, tests a property of an asset such as nulls in a column or a schema, and the result appears in the UI with a severity. Setting blocking=True stops downstream assets from materializing when the check fails, which turns a check into a gate. By default runs continue past a failed check.

Datadog Data Jobs Monitoring covers jobs across Spark on Kubernetes, EMR, Dataproc, Databricks, Airflow, dbt, AWS Glue and Azure Data Factory. It detects failed and slow jobs and data skew, and compares runs. It works at the job level and does not read table contents. Pick this layer first when your incidents are failed or late jobs, and add a table layer when they're wrong numbers behind a green run.

Elementary is the dbt-native option. It's Apache-2.0 licensed, has 2.4k GitHub stars as of 2026-11-03, and runs anomaly detection as dbt tests, with monitors for freshness and volume plus schema changes. The open-source dbt package and CLI are self-managed, and a hosted Elementary Cloud sits on top. It fits a team whose models already live in dbt, because the checks are configuration in the same project.

Great Expectations (Apache-2.0, 11.9k stars as of 2026-11-03) and Soda Core express checks as code. Soda Core verifies data contracts inside your pipelines, and Soda's documentation states that it does not include observability features. Metric monitoring and anomaly detection arrive with Soda Cloud, a managed service. That split is common: the open-source piece enforces rules you know, and the hosted piece learns what normal looks like.

For lineage and catalogs, the OpenLineage documentation describes an open framework in the LF AI & Data Foundation, built on datasets, and on the jobs and runs that touch them, extended by facets, with integrations for Airflow, Spark, Flink and dbt, as well as Great Expectations. Marquez is its reference implementation. OpenMetadata adds data quality tests and a profiler, plus incident management and alerts, in 1 catalog. Open-source stacks give full control and ask you to host and upgrade every piece and connect them yourself.

Both warehouses ship a first layer. Snowflake system data metric functions measure common metrics without configuration, run on a schedule that defaults to once every hour, and require Enterprise Edition. Its data quality documentation says anomaly detection currently covers volume and freshness, billing is serverless compute for scheduled runs, and an account holds up to 50,000 function associations.

The schedule sets your detection delay. A freshness check that runs every 60 minutes on a table with a 240 minute SLA can alert up to 300 minutes after the last good load: the allowance plus 1 check interval. Tighten the schedule on the few tables where that gap is too long, and accept the default elsewhere.

Databricks data quality monitoring has 2 parts. Anomaly detection checks freshness, using a model of each table's commit history, and completeness, whether the last 24 hours brought roughly the expected rows. Data profiling tracks summary statistics and drift; it was formerly called Lakehouse Monitoring. Both cover Unity Catalog tables and bill as serverless compute. That's enough while all of your data lives on 1 platform, but the checks stop at its edge.

Platforms connect to your warehouses and orchestrators, as well as BI tools, and monitor tables from outside. Monte Carlo's documentation lists table monitors for freshness and volume and for schema changes, metric monitors with anomaly detection or fixed thresholds, validation monitors with custom SQL, and job and query performance monitors. Monitors can be defined as code and deployed through CI/CD, and a Terraform provider manages account setup.

Deployment varies and deserves a question in any evaluation. Monte Carlo documents an agent that runs in your cloud, a Snowflake Native App and private network links, plus a dedicated instance and EU hosting. Bigeye describes itself as an enterprise data observability platform covering pipeline and data quality monitoring, with schema change detection and lineage. Ask each vendor what metadata leaves your environment and which of these options your security team accepts.

The category also consolidates. On April 23, 2025, Datadog announced it was acquiring Metaplane, which it described as an end-to-end data observability platform with machine learning monitoring and column-level lineage, and said Metaplane would continue as Metaplane by Datadog. Check current packaging before you assume which products ship together. Prices change often and are left out here, so ask for them in writing.

No tool here is the right answer in general, so the decision runs through your constraints. Start from where incidents come from. If runs fail or crawl, begin with job monitoring. If runs are green and numbers are wrong, begin with table checks, in the system that already holds the data.

Then count how many places the data lives. One warehouse favors its native checks. A dbt project favors Elementary or dbt tests. Several warehouses and a lake feeding BI tools in the same incident favor a platform, because only a tool that sits outside them can follow a break from one system into the next. Last, weigh who maintains it: open-source stacks cost engineer time, while platforms cost money and a metadata-sharing review.

Replay your worst incident from the last quarter instead of watching a demo. Load the affected tables, ask the tool how soon it would have alerted, and check whether the alert named the upstream cause. A tool that detects the break but can't place it still leaves you doing the investigation by hand.

Test 4 things on your own stack. First, coverage: does it connect to every system in the incident. Second, alert quality: how many alerts fire for 1 upstream break, and can they group by shared cause. Third, delay: what the check schedule adds to detection time. Fourth, ownership: who tunes thresholds once the trial ends. A tool that answers all 4 for your tier 1 tables is a candidate whatever its category.

Interviewers rarely ask you to name a vendor. They ask how you would know a pipeline you designed is broken and how you would find the cause, then listen for whether you separate job health from data health and where lineage fits. The data pipeline interview questions cover that ground, and the data pipeline practice problems let you design the pipelines these checks watch.

The data engineering system design round is where monitoring choices get probed hardest, and dbt-based answers lean on dbt interview questions. For the concepts underneath the tools, read what data observability is, data quality tools and data lineage tools.