OpenMetadata

OpenMetadata ingests metadata from 50+ data sources (databases, data warehouses, BI tools, data lakes, pipelines) through a pluggable connector architecture. Each connector implements a standardized extraction interface that maps source-specific metadata schemas to OpenMetadata's unified entity model, with support for incremental ingestion, scheduling via Airflow, and automatic lineage extraction during the ingestion process.

OpenMetadata tracks data lineage at column granularity by parsing transformation logic from SQL, dbt, Spark, and pipeline definitions, building a directed acyclic graph (DAG) of column dependencies across tables and systems. The lineage engine reconstructs column-to-column transformations, enabling impact analysis and root cause investigation across the entire data stack with interactive UI visualization.

OpenMetadata provides a Java SDK that enables developers to programmatically query, create, and update metadata entities, execute lineage analysis, and manage access control. The SDK handles authentication, serialization, and API communication, providing a type-safe interface to the OpenMetadata REST API with support for batch operations and streaming responses.

OpenMetadata provides a Kubernetes operator that automates deployment, scaling, and lifecycle management of OpenMetadata components (backend service, ingestion scheduler, search cluster) on Kubernetes. The operator manages configuration, database migrations, and service dependencies, enabling declarative infrastructure-as-code deployment with automatic reconciliation.

OpenMetadata supports bulk import and export of metadata entities (tables, columns, glossary terms, owners) via CSV and JSON formats, enabling migration from other metadata platforms, backup/restore workflows, and integration with external metadata sources. The import process validates schemas, handles duplicates, and provides detailed error reports for failed records.

OpenMetadata's data profiler analyzes table and column statistics (row count, null percentage, cardinality, min/max, distribution histograms) on a schedule and stores historical trends. The profiler integrates with the ingestion framework to run after data loads, enabling detection of data quality anomalies through statistical comparison with historical baselines.

OpenMetadata profiles table and column statistics (null counts, cardinality, distribution, data types) and executes parameterized data quality tests (null checks, uniqueness, range validation, custom SQL assertions) on a schedule. Test results are stored with historical trends, enabling detection of data quality regressions and integration with data observability workflows through event-driven notifications.

OpenMetadata enables teams to define data contracts (schema, quality SLAs, ownership, update frequency) as versioned metadata entities, attach semantic annotations (business glossary terms, tags, descriptions) to tables and columns, and enforce contract compliance through automated validation. Contracts are queryable and can be integrated into CI/CD pipelines to prevent breaking changes to data assets.

OpenMetadata indexes metadata entities (tables, columns, dashboards, glossary terms) using Elasticsearch or OpenSearch with full-text search, and optionally generates vector embeddings of descriptions and metadata to enable semantic similarity search. Users can search by natural language queries (e.g., 'customer revenue metrics') and receive ranked results based on relevance, with faceted filtering by owner, domain, and data type.

OpenMetadata enforces role-based access control (RBAC) at the entity level (table, column, dashboard) with support for custom roles and permissions. Access policies can be defined based on data lineage — for example, granting read access to all downstream tables when a user has access to an upstream source — enabling permission inheritance through the data pipeline.

OpenMetadata enables teams to claim ownership of data assets (tables, dashboards, domains), add descriptions and documentation, and collaborate through comments and activity feeds. Ownership is tracked at the entity level with support for multiple owners, and changes to assets trigger notifications to owners and stakeholders, creating accountability and enabling self-service metadata management.

OpenMetadata exposes a Model Context Protocol (MCP) server that allows LLMs and AI agents to query metadata, execute lineage analysis, and retrieve data contracts through a standardized interface. The MCP server handles authentication, context enrichment, and response formatting, enabling natural language queries like 'show me all tables owned by the finance team with PII data' to be executed against the metadata catalog.

OpenMetadata provides a hierarchical domain structure for organizing data assets by business area, and a glossary system for defining business terms with relationships (synonyms, parent/child, related terms). Glossary terms can be linked to table and column metadata, enabling semantic understanding of data and supporting data governance through standardized business vocabulary.

OpenMetadata publishes events (entity created, updated, deleted, lineage changed, quality test failed) to an event bus (Kafka, webhook) that external systems can subscribe to. This enables real-time metadata synchronization with downstream tools, triggering workflows when data assets change, and maintaining eventual consistency across the data stack without polling.