OpenMetadata

OpenMetadata implements a centralized metadata store using a typed entity model (databases, tables, columns, dashboards, pipelines, etc.) persisted in PostgreSQL/MySQL with REST API access. The Entity Management and Repository Layer provides CRUD operations on metadata entities with version control, lineage tracking, and relationship management through a schema-driven approach that enforces consistency across all ingested metadata sources.

OpenMetadata tracks data lineage at column granularity by parsing SQL queries, ETL job definitions, and pipeline DAGs to build a directed acyclic graph (DAG) of data transformations. The Lineage and Domain Management system stores lineage edges in the metadata repository and exposes them via REST APIs and UI visualizations, enabling users to trace data provenance from source to sink and identify downstream impact of schema changes.

OpenMetadata provides Kubernetes Operator and Helm charts for cloud-native deployment, enabling declarative infrastructure-as-code management of OpenMetadata instances. The deployment architecture supports horizontal scaling of the OpenMetadata service (stateless), with external PostgreSQL/MySQL and Elasticsearch/OpenSearch backends. The Kubernetes Operator automates upgrades, configuration management, and backup/restore operations, enabling GitOps-based deployment workflows.

OpenMetadata's Data Profiler computes statistical profiles for tables and columns (null counts, cardinality, min/max values, distribution histograms, correlation analysis) by executing SQL queries against source systems. Profiles are stored as metadata and tracked over time, enabling trend analysis and detection of statistical anomalies (e.g., sudden increase in null values, unexpected cardinality changes). The profiler integrates with data quality tests to provide context for quality issues.

OpenMetadata's Metadata Ingestion Framework provides a plugin-based architecture for extracting metadata from diverse sources (databases, data warehouses, BI tools, data lakes, orchestration platforms). Each connector implements a standardized interface to extract entities, relationships, and lineage, transform them into OpenMetadata's entity model, and load them into the central repository. The framework supports both batch ingestion (scheduled jobs) and event-driven ingestion via Airflow, Kafka, or direct API calls.

OpenMetadata's Data Profiler and Quality Validations system automatically computes statistical profiles (null counts, cardinality, distribution, min/max values) for tables and columns on a schedule, and executes user-defined data quality tests (e.g., 'column X should have <5% nulls', 'column Y values must match regex pattern'). Test results are stored as metadata entities linked to tables, enabling trend analysis and alerting on quality degradation. The system integrates with dbt tests, Great Expectations, and custom SQL validators.

OpenMetadata indexes all metadata entities (tables, columns, dashboards, pipelines, glossary terms) into Elasticsearch or OpenSearch, enabling full-text search with relevance ranking and faceted filtering by entity type, owner, domain, tags, and custom attributes. The Search and Indexing system uses BM25 scoring for relevance and supports advanced queries (wildcards, boolean operators, field-specific searches). Search results are ranked by relevance and enriched with lineage, ownership, and quality metadata.

OpenMetadata implements fine-grained RBAC through the Authentication and Authorization system, supporting multiple auth providers (OAuth2, SAML, LDAP, custom) and role definitions (Admin, DataSteward, DataConsumer, etc.). Access control is enforced at entity level (who can view/edit specific tables, columns, dashboards) and operation level (who can approve data quality tests, manage glossaries). The system integrates with governance workflows (approval chains, ownership assignment, domain management) to enforce data stewardship policies.

OpenMetadata provides collaborative features for teams to enrich metadata with descriptions, tags, glossary terms, and custom attributes. The Glossary and Domain Management UI enables creation of business glossaries with term hierarchies, definitions, and relationships to data assets. The Activity Feed and Rich Text Editor track all metadata changes with user attribution, enabling teams to discuss data assets, ask questions, and resolve ambiguities through inline comments and mentions.

OpenMetadata exposes its metadata repository and capabilities through an MCP (Model Context Protocol) server, enabling AI agents and LLMs to query metadata, execute searches, retrieve lineage, and access data quality information via standardized MCP tools. The MCP Server and Java SDK (implemented in openmetadata-mcp module) provides authentication-enriched context extraction, allowing AI systems to respect OpenMetadata's RBAC policies while accessing metadata. This enables natural language queries over metadata ('show me all tables owned by the analytics team with quality issues') and AI-assisted data discovery.

OpenMetadata supports definition and tracking of data contracts (agreements about data quality, freshness, and availability) and SLAs for data products. Data contracts are defined as metadata entities linked to tables/datasets, specifying expected quality metrics, update frequency, and ownership. The system tracks contract compliance by comparing actual data quality metrics (from profiling) against contract expectations, enabling data teams to validate that data products meet their promised SLAs.

OpenMetadata's Event System and Workflows enable real-time metadata updates through event streaming (Kafka, webhook) and trigger-based workflows. When metadata changes occur (table added, quality test fails, ownership changes), events are published to configured webhooks or Kafka topics, enabling downstream systems to react. The system supports custom workflows that can execute actions (send notifications, update external systems, trigger data pipelines) based on metadata events.