Data Lineage Tools
Open Source and Column-Level Options (2026)
The main open source data lineage tools are OpenLineage with Marquez, the SQL parsers SQLLineage and SQLGlot, and the platforms DataHub and OpenMetadata. Databricks and Snowflake build column-level lineage into their platforms, and so does BigQuery. They capture lineage by different routes (job events or parsed SQL, or the platform's own query history), so the right one depends on where your transformations run and who reads the graph.
What are data lineage tools?
Data lineage tools record where a dataset or column came from, which job produced it, and what consumes it next. That record answers 2 questions a data engineer gets every week: what breaks if I change this column, and where did this number come from. The open source options are OpenLineage with its Marquez server, the SQL parsers SQLLineage and SQLGlot, and the metadata platforms DataHub and OpenMetadata. Databricks Unity Catalog and Snowflake ship column-level lineage inside their own platforms, and so does BigQuery.
The names do not all mean the same kind of thing, which is why comparisons of them mislead. OpenLineage is a standard for collecting lineage metadata. Clients and integrations emit it. Marquez is a server that receives those events. DataHub and OpenMetadata are metadata platforms that store a graph and put search and ownership beside it, with a glossary on top, and both can take lineage from several routes. SQLLineage and SQLGlot are libraries that read SQL text. These layers combine more often than they compete: Google's lineage service imports OpenLineage events and OpenMetadata accepts them; DataHub's Airflow plugin builds on Airflow's OpenLineage support.
A lineage graph also serves a second job in data engineering work. It is the map you walk upstream during an incident, the one described in data observability, and the blast radius you read before a schema change. Data lineage as a concept, and the data catalog that usually hosts the graph, have their own guides.
Know Data Lineage Tools the way the interviewer who asks it knows it.
How lineage tools capture column lineage
Every tool gets its column edges by one of 3 routes, and the route decides what it can see. One route is events from the running job. The OpenLineage Spark integration walks the Spark logical plan and sends a columnLineage facet, and its documentation says this is on by default. It reads what Spark actually executed, so it sees through DataFrame code that no SQL parser could read. Its documented limit is JDBC sources, where a SQL parser reads only the query text and collects no field types or transformation types.
Another route is parsing SQL text, which needs no access to the running system. SQLLineage prints column paths from a file with sqllineage -f test.sql -l column, and does not expand wildcards unless you give it metadata. SQLGlot's lineage function builds a column graph from a query plus a schema. dbt's column-level lineage relies on SQL parsing as well, and its documentation lists the failures: ambiguous or complex SQL, Python models that cannot be parsed, and hard-coded table names written instead of ref. OpenMetadata parses query logs. It tries SQLGlot first and falls back to SQLFluff and then SqlParse. It keeps only statements that move data: CTAS, INSERT ... SELECT, MERGE and view definitions.
A third route is the platform's own query history. Databricks captures lineage automatically for queries run on Databricks, and Snowflake records it between table-like objects. There's nothing to install, and nothing outside the platform is visible. If your pipeline touches both Spark and a warehouse, or adds a streaming system, expect to combine routes.
How OpenLineage describes a column edge
fieldsinputFieldsdatasetDIRECTINDIRECTIDENTITY, TRANSFORMATION, AGGREGATIONJOIN, GROUP_BY, FILTERSORT, WINDOW, CONDITIONALDirect vs indirect edges: a worked example
The OpenLineage column lineage facet separates a column that supplies a value from a column that only shapes the result. Take one query that returns 2 columns.
select
o.order_id,
sum(l.qty * l.unit_price) as revenue
from orders o
join order_lines l on l.order_id = o.order_id
where o.status = 'paid'
group by o.order_idorder_id is a DIRECT, IDENTITY copy of orders.order_id. revenue is a DIRECT AGGREGATION of order_lines.qty and order_lines.unit_price. That is 3 direct edges. The join reads orders.order_id and order_lines.order_id, the where reads orders.status, and the group by reads orders.order_id again. That is 4 indirect edges, 7 in all.
The indirect edges are the ones a select-list view drops. dbt's documentation says its column-level lineage reflects select statements and not other usage such as joins and filters, so a view of that kind draws 3 of 7 edges and hides 57% of the dependencies. The cost shows in impact analysis. Change the meaning of orders.status and a direct-only graph reports no affected output, while the filter edge reaches all 2, because a filter changes which rows exist for every column. When you compare tools, count edges against a query you have traced by hand.
Open source data lineage tools, one by one
OpenLineage is an Apache-2.0 project, at graduate stage in the LF AI & Data Foundation, with integrations that cover Spark and Flink as well as Airflow and dbt. Marquez, also Apache-2.0, is an open source metadata service that uses OpenLineage to collect metadata and draws dataset and job lineage. Its README does not mention column-level lineage, so check the version you deploy before you count on column edges from it. Choose this pair when the goal is a vendor-neutral event stream that more than one backend can read.
DataHub is Apache-2.0, with column-level lineage in its README feature list. Its Airflow plugin uses DataHub's SQL parser, which uses DataHub's metadata to generate column-level lineage, and the parser handles multi-statement SQL only when enable_multi_statement_sql_parsing is switched on. Manually declared inlets and outlets are table-level, so column lineage for a custom operator needs a custom extractor. Choose it when you want discovery and ownership in the same platform as lineage and your orchestration is Airflow.
OpenMetadata is Apache-2.0, with 130+ connectors and column-level lineage among its capabilities. It builds lineage from query logs and view definitions and accepts OpenLineage events. You can also edit table and column lineage by hand or through its REST API and Python SDK. Choose it when most of your lineage comes from warehouse query history and you want to patch the gaps manually.
SQLLineage is MIT-licensed and prints column paths from SQL files. SQLGlot is MIT-licensed, supports more than 30 dialects and its lineage function takes the SQL plus a schema. Choose these when lineage is a check in CI or a feature of your own tool and a server would be overhead. Spline is the Apache-2.0 option for Spark: its agent attaches as a query execution listener, and by default it tracks only persistent writes. Its documentation page does not state column-level detail, so verify that before choosing it for column lineage.
Open source lineage tools side by side
- License
- What it is
- Column lineage
- Captured by
- Needs a server
- LicenseApache-2.0
- What it isSpec and server
- Column lineageFacet, per integration
- Captured byJob events
- Needs a serverMarquez server
- LicenseApache-2.0
- What it isMetadata platform
- Column lineageListed as a feature
- Captured byIngestion, SQL parser
- Needs a serverYes
- LicenseApache-2.0
- What it isMetadata platform
- Column lineageListed as a feature
- Captured byQuery logs, events
- Needs a serverYes
- LicenseMIT
- What it isPython library
- Column lineageCLI, column mode
- Captured byParsing SQL text
- Needs a serverNo
Column-level lineage in dbt projects
dbt gives you lineage between models from ref and source calls with no extra tooling, and column-level lineage as a second layer. Its documentation describes 2 places to see it: Catalog in the dbt platform, which requires an Enterprise or Enterprise+ plan, and local development, which requires strict static analysis and is read with the column_lineage info option of dbt show. It labels a column Passthrough when it is reused unchanged and Rename when it's only renamed, and it shows the formula for a transformed one.
If you run dbt Core on a warehouse, the open source route is OpenLineage. openlineage-dbt supports 16 adapters, including BigQuery, Snowflake and Redshift, as well as Postgres, Databricks and DuckDB. It emits a columnLineage facet only when column-level parsing is enabled. A catalog such as DataHub or OpenMetadata can then show it next to the models. These configurations come up in dbt interview questions.
Column-level lineage built into Databricks, Snowflake and BigQuery
| Platform | What it captures | How long it keeps it | Documented limits |
|---|---|---|---|
| Databricks Unity Catalog | Column lineage automatically for queries run on Databricks, in SQL and the DataFrame API | Indefinitely in Catalog Explorer, a rolling 1 year in system tables | No column lineage in the UI for path references, none for UDFs, and MERGE, UPDATE and DELETE are not reflected |
| Snowflake | Column lineage between any 2 table-like objects, on Enterprise Edition or higher | 1 year | Shared databases and INFORMATION_SCHEMA objects are excluded, and viewing needs the VIEW LINEAGE privilege |
| BigQuery, through Google Cloud lineage | Column lineage for BigQuery jobs once the Data Lineage API is enabled | 30 days | No column lineage for load jobs or routines, jobs with over 1,500 column links fall back to table level, and only top-level columns are covered |
Where column-level lineage goes missing
Every route has a blind spot, and the blind spots decide which tool fits. Parsers fail on SQL they cannot read, and the failure can be quiet: wildcards stay unexpanded without metadata, and a table named by a string instead of a reference never links. Event-based capture sees only jobs that were instrumented, so a script run by hand leaves no edge. Platform logs see only that platform, and each documents statement types it skips.
Silent downgrades are another gap. A tool that cannot resolve a column mapping can fall back to table level without saying so, as Google's lineage does for a BigQuery job over 1,500 column links. A graph that looks complete at table level can then have no column edges for exactly the wide, heavily transformed tables where you need them. Ask each tool how it reports a statement it couldn't parse.
Another gap is what the graph means. A tool that draws only select-list edges answers where a value came from but not what else a column affects, and one that mixes the 2 without labels makes impact analysis noisy. Prefer tools that keep the DIRECT and INDIRECT distinction visible.
Which data lineage tool fits which situation
If your transformations run almost entirely inside 1 warehouse, its built-in lineage is the smallest step: nothing to deploy, though you accept that platform's retention and statement limits. If the work spans Spark jobs and an Airflow and dbt setup, an event standard such as OpenLineage keeps instrumentation neutral, and a backend can change later without re-instrumenting.
If the people asking lineage questions are analysts and owners rather than the platform team, put the graph next to search and ownership with DataHub or OpenMetadata, at the cost of running a platform. If lineage is a CI check, such as failing a pull request that drops a column something reads, a parser library is enough and a server is overhead. If your SQL is in dbt and the Enterprise plan is available, its Catalog shows column lineage with no extra system, while a project on Core can emit events through openlineage-dbt.
None of these choices is permanent, and most mature setups run 2 of them. Practicing the reasoning matters more than the tool name in a loop: data pipeline interview questions and data pipeline practice problems cover how to trace and defend a pipeline, and the system design round is where an interviewer asks how you would find what a change breaks.
How to test a lineage tool on your own queries
Pick 4 queries from your own warehouse: a join with a filter and a group by, a MERGE, a query with a wildcard, and one that calls a user-defined function. Write the expected direct and indirect edges for each by hand, so you hold a number to compare against.
Run each tool on the same 4 and count the edges it finds against your list. Note what it misses: edges of either kind, or whole statements. Finally feed it 1 query it can't parse and see whether it flags the gap, drops to table level or shows nothing. A tool that fails loudly is easier to trust than one that fails with a clean-looking graph.
Data lineage tools FAQ
What are the best open source data lineage tools?+
Which tools support column-level lineage?+
Is OpenLineage a tool or a standard?+
What is the difference between table-level and column-level lineage?+
How long do the platforms keep lineage?+
Why is my column lineage incomplete?+
The candidate who gets the offer
- 01
Reading a solution is not the same as writing one
Every engineer who has frozen on a query they had read a dozen times knows the gap. The only preparation that closes it is producing the answer yourself, under time, before the interview does it for you
- 02
76% of hiring managers reject on the coding task, not the resume
From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice
- 03
System design comes down to the calls you defend out loud
Ingestion, batch vs streaming, the bronze/silver/gold layers, idempotency, backfill and replay. Sketching the pipeline and naming the failure modes is the signal, not the boxes
Related guides
Data pipeline interview questions
The pipeline design and operations questions interviewers ask
Data pipeline practice problems
Pipeline design problems to practice before the loop
Data engineering system design round
What the design round asks and how to prepare for it
dbt interview questions
Questions on dbt models and on tests and sources
Data lineage explained
The concept, the questions it raises in interviews and a deeper dive
What is data observability
How lineage turns scattered alerts into 1 incident
Data catalogs
The tools that usually host the lineage graph
Compete in the Weekly
Dirty, production-shaped data, scored blind each week.