Data Catalog Tools

DataHub is built around an event stream. Its metadata service stores entities and publishes every metadata change to Kafka, which feeds the search index and the graph, and a Python ingestion framework run from a YAML file pushes metadata in through Kafka or the REST API. The repository has about 12.9k stars as of 2026-11-03, its latest release is v1.8.0 (2026-10-22), and its maintainers also sell a hosted version, DataHub Cloud. It fits a team that already runs Kafka, wants to push metadata from its own pipelines and doesn't mind operating several stateful services.

OpenMetadata centres on a UI. An admin registers a service and picks one of its 130+ connectors. After testing the connection, they schedule agents for metadata and usage, for lineage and dbt, and for profiling. The repository has about 15.7k stars, and the latest release is 2.0.7 (2026-10-27). It fits a team that wants analysts and stewards to set up sources without writing ingestion code. It also ships data quality tests and a profiler in the same product.

Apache Atlas comes from the Hadoop world. It offers a type system with classifications, plus lineage, search and security. Its hooks cover Hive, HBase, Sqoop and Kafka, and the store behind it is JanusGraph with HBase and Solr. It fits an organisation whose estate is still Hive and Hadoop and which wants classification tags to drive access policy. Outside that estate, the hooks have little to hook into.

Amundsen, a search-first catalog with a Flask frontend, an Elasticsearch search service and a metadata service on Neo4j or Apache Atlas, was archived in September 2026 for inactivity. Its repository (about 4.8k stars) is read-only. Existing installs keep working, but a new evaluation should start elsewhere, since nobody will fix a bug or add a connector.

Unity Catalog is the fourth open source name that turns up, and it is an engine catalog. The open source server (about 3.6k stars, latest release v0.7.0) is a sandbox project of the LF AI & Data Foundation. It manages tables, files, functions and AI models, and it's compatible with the Hive Metastore API and the Iceberg REST catalog API. Apache Polaris plays the same role for Iceberg tables. Both answer where a table lives. Neither replaces a place where an analyst searches for the right one.

Each large platform now ships a catalog, and for data that lives on that platform it is often the first one to evaluate because there's nothing to install. The trade is reach: each catalog sees its own platform best and everything else through connectors.

The AWS Glue Data Catalog stores the location and schema of your data, plus runtime metrics. Crawlers scan sources to populate it, or you define tables by hand. Athena, Lake Formation, EMR and Redshift Spectrum all read it. It's the default engine catalog for lakes on S3, and it offers column statistics, lineage records and, in preview, glossary terms.

Databricks Unity Catalog governs data and AI assets under a catalog.schema.object namespace. It enforces privileges and row and column filters, and it tracks lineage; the Catalog Explorer handles search. Snowflake Horizon Catalog adds column-level lineage across Snowflake, external databases and BI tools. It also classifies sensitive data and exposes an Iceberg REST catalog API, so Spark, Flink and Trino can read Snowflake-managed Iceberg tables. Google Cloud's Knowledge Catalog, formerly Dataplex Universal Catalog, harvests BigQuery and Looker automatically, along with Pub/Sub, Cloud Storage and other sources. It attaches aspects such as schema, contacts and data quality, with a business glossary on top.

Microsoft Purview pairs a Data Map with a Unified Catalog inside a wider data security and compliance suite. It fits an organisation that already governs its data through Microsoft tooling and wants one place for sensitivity labels and discovery. Whichever platform you run, check how far the catalog sees outside it before assuming it covers the whole estate.

Atlan and Collibra are commercial catalogs that run as hosted products, and so is Alation. A commercial catalog, or a hosted version of an open source one such as DataHub Cloud, mostly buys you someone else carrying the operations. That means upgrades and scaling the stores underneath. It also covers connector maintenance and gives you an escalation path when ingestion breaks at 2 a.m.

That makes the decision a staffing question as much as a feature one. A platform team of 2 that already runs Kubernetes and Kafka can absorb DataHub. A team of 2 that also owns every dashboard may spend more on upkeep than the license would cost. Count the engineer hours a self-hosted catalog takes per month against a quote, using your own connector list rather than a vendor demo.

Start from coverage, not features. List the 10 sources your analysts query and your engineers touch most: the warehouse, the lake, the orchestrator, the transformation tool, the BI tool and event streams. A catalog is worth adopting only if it ingests most of them with maintained connectors, since an asset missing from the catalog sends people back to asking in chat.

Then test lineage on a real table. Lineage is only as complete as what feeds it, and the sources differ: parsed query logs, a transformation tool such as dbt, or run events in the OpenLineage format. Pick 1 dashboard and check that the catalog walks back from it to the raw source, including the hop through your orchestrator. A gap at 1 hop breaks every upstream walk past it.

Check ownership and freshness of the metadata itself. A catalog full of tables with no owner and a 2-year-old description is worse than none, because it looks authoritative. Look for owner assignment from the source, such as a dbt meta field or a warehouse tag, and for a way to flag stale or deprecated assets.

Last, price the operations. List every service the catalog needs to run, who patches each one and what happens to search while an upgrade runs. For a self-hosted catalog this list is the real bill, and for a platform catalog it is zero but the coverage ends at the platform's edge.

A catalog is one tool in a metadata stack. A transformation tool such as dbt produces table lineage from its model references, and teams that use it often want to see dbt interview questions before they decide where documentation should live. A lineage collector records which job read and wrote which dataset. OpenLineage is the open specification for those records. It defines entities for datasets, jobs and runs, with facets that add detail, and it's a graduate project of the LF AI & Data Foundation. It has integrations for Airflow, Spark, Flink, dbt and Trino, among others. Marquez is its reference implementation for storing and browsing the events.

Schema registries hold the contracts for event streams. Data quality and observability tools measure freshness and volume and check values, then feed their results into the catalog so a table's page shows whether it is healthy right now. The deeper guides cover each: data lineage goes into how the graph is built, data observability covers the checks, and data governance covers the policies a catalog helps enforce.

Treat the catalog as the place where those signals meet people. It rarely generates them. A team that buys a catalog hoping it will create ownership and lineage, and quality checks, by itself usually gets an empty index, and the fix is upstream, in the tools that emit the metadata.

Catalog questions rarely ask you to name a product. They appear inside design questions: how would analysts discover the tables this pipeline produces, how do you know who to page when a source breaks, how do you stop people building on a deprecated table. The answer is metadata. Record ownership and lineage where people search, show freshness there too, and draw a clear line between what's automatic and what a steward must write.

A strong answer separates the 2 catalogs. Say that engines need an engine catalog on the query path and that people need a discovery catalog beside it, then say which signals feed the second. The data pipeline interview questions and data pipeline practice problems cover the design side, and the system design round guide shows how interviewers probe operability.