Datahub

DataHub is a metadata management system and data catalog platform designed to provide a centralized directory for discovering, managing, and documenting datasets across a diverse data stack. It serves as a comprehensive framework for metadata management, incorporating a data governance framework to classify sensitive information and assign ownership for organizational accountability.

The platform distinguishes itself through AI-enabled data discovery, which connects large language models to a metadata graph to allow for natural language search and exploration of data assets. It also provides specialized data lineage tools that map column-level dependencies to track the flow of data from source to consumption.

The system covers a broad range of capabilities including universal metadata search, data quality monitoring for schema drift and freshness, and dataset profiling. It utilizes a plugin-based ingestion framework to automate the extraction of schemas and usage metrics from warehouses and business intelligence tools.

Features

Data Catalogs - Acts as a centralized directory for discovering, managing, and governing diverse organizational data assets.

Data Catalogs - Serves as a centralized directory for discovering, managing, and documenting datasets and metadata across a diverse data stack.

Metadata-to-Agent Bridges - Provides a protocol that bridges the metadata graph to AI agents for natural language data discovery.

Data Lineage - Tracks the origin and transformation history of data assets to understand upstream and downstream dependencies.

Natural Language Data Exploration - Uses NLP and metadata graphs to generate insights and explore datasets through natural language.

Column-Level Lineage Extraction - Maps granular column-level dependencies to track the flow of data from source to consumption.

AI-Powered Exploration - Integrates large language models with a metadata graph to enable natural language search and discovery.

Data Governance - Provides frameworks for managing data policies, classifying sensitive info, and assigning organizational ownership.

Database Metadata Ingestion - Automates the extraction of technical and operational metadata from external warehouses and BI tools.

Enterprise Metadata Management - Provides a platform for ingesting and querying technical and operational metadata from warehouses and BI tools.

Metadata Knowledge Graphs - Stores data assets and their relationships as a directed graph to facilitate complex lineage tracking.

Metadata Search Engines - Implements a search interface to index and query structured metadata for precise retrieval of datasets across the data stack.

Data Asset Lifecycle Management - Tracks data assets and their ownership throughout their entire lifecycle to ensure organizational oversight.

Pipeline Metadata Extraction - Automates the extraction of schemas and usage metrics from warehouses and BI tools.

Data Quality Monitors - Provides a tracking system for monitoring data freshness, volume changes, and schema drift to ensure ecosystem reliability.

Data Quality Profilers - Generates metadata profiles including schemas, data statistics, and technical documentation for individual datasets.

Metadata Indexing - Utilizes Elasticsearch to index metadata entities for high-performance full-text search and filtering.

Metadata Schema Extensions - Implements a flexible metadata structure that allows custom entity types and attributes without a rigid schema.

Metadata Querying - Offers programmatic interfaces and SDKs for retrieving and updating catalog information and custom properties.

REST APIs - Provides a standardized REST API for programmatically managing catalog entities and properties.

Data Ingestion Plugins - Employs a modular plugin framework of connectors to ingest metadata from various external sources.

Metadata Event Processors - Processes data asset changes via an asynchronous event stream to keep the metadata graph consistent.

Data Catalog - Metadata search and discovery tool.

Repository and Developer Analysis - Generalized metadata search and discovery for organizational data.

linkedindatahub

Features

Star history