apache/gravitino

Features

• Unified Metadata Catalogs - Provides a unified metadata catalog that aggregates metadata across diverse data sources using a standardized property system.
• Data Catalogs - Creates catalogs for relational, messaging, and model types with connection validation.
• Federated Metadata Governance - Unifies metadata from diverse databases, lakehouses, and messaging systems into a single interface for centralized governance.
• Unified AI Asset Management - Unifies the management of AI models and version lineages alongside traditional tabular data and files.
• AI Model Management - Provides systems for registering, modifying, and maintaining the lifecycle of AI model definitions.
• Model Lineage Trackers - Tracks and versions the evolution and provenance of machine learning models throughout their lifecycle.
• Metadata Tagging - Associates, disassociates, and searches for metadata items based on custom organizational tags.
• Iceberg Metadata Services - Provides a metadata service for Iceberg tables via Hive Thrift, JDBC, and REST APIs.
• Compute Engine Integrations - Connects centralized metadata to various processing engines to enable unified data access.
• Catalog Proxying - Provides a unified metadata access layer by acting as an intermediary proxy for various catalog backends.
• Request Proxying - Forwards Iceberg REST API operations to backends such as Hive, JDBC, or other REST catalog servers.
• REST Catalog Provisioning - Serves as a REST-based catalog for Iceberg tables to enable dataset access sharing across compute engines.
• Multi-Provider Object Storage - Connects to S3, GCS, ADLS, OSS, and HDFS to provide unified storage and retrieval of data.
• Cross-Source Querying - Executes single SQL queries that join datasets across different data stacks and diverse sources.
• Data Source Unification - Organizes metadata from diverse sources into a hierarchical structure of metalakes, catalogs, and schemas.
• Table Schemas - Provides definitions and operations for creating structured data tables with custom columns and distribution settings.
• Logical Groupings - Organizes tables and assets within a catalog using named logical groupings.
• Database Schema Management - Creates and removes database schemas within the metadata lake to organize and namespace assets.
• Distributed System Metadata - Links distributed dataframes to a unified metadata system for retrieving tables and filesets from diverse sources.
• Federated Data Query Engines - Accesses data across multiple catalogs through a unified interface with consistent access control.
• Lakehouse Metadata Coordination - Coordinates metadata for file system-based lakehouse architectures to provide a unified discovery interface.
• Lakehouse Schema Management - Creates, modifies, and deletes schemas and associated properties within a lakehouse catalog.
• Federated Metadata Lakes - Serves as a unified metadata layer managing tables, files, and AI models across diverse data sources and cloud storage.
• File Asset Management - Defines managed or external storage locations and handles metadata for file-based data assets.
• File Storage and Metadata Management - Manages schemas and filesets across local filesystems, HDFS, and cloud object storage.
• Hybrid Metadata Management - Manages relational tables and file-based datasets as a single source for downstream applications.
• Iceberg REST Catalog Servers - Provides a standalone server implementing the Iceberg REST specification for view and scan management.
• Metadata Caches - Stores table metadata in memory to reduce lookup latency and validate storage locations.
• Metadata Orchestration - Coordinates schemas and table definitions for file-based lakehouse architectures across cloud and on-premises storage.
• Metadata Asset Retrieval - Retrieves lists and detailed metadata for catalogs, schemas, tables, models, and filesets across diverse sources.
• Metadata Caching - Caches frequently accessed metadata in memory to reduce database load and improve response latency.
• Query Performance Tuning - Accelerates data retrieval using primary keys and data-skipping indexes like Bloom filters and MinMax.
• Schema Management - Provides tools for programmatically creating, modifying, and removing schemas within metadata catalogs.
• Programmatic Table and Index Managements - Enables programmatic creation and deletion of relational tables with custom partitioning and indexing.
• AI Artifact Catalogs - Maintains structured catalogs for AI-specific artifacts, including model metadata and governance.
• Catalog Interfaces - Serves as a REST catalog for specific table formats to standardize metadata access.
• Cloud Storage Integrations - Connects to S3, GCS, OSS, and Azure Blob Storage using the Hadoop protocol.
• Iceberg REST Specifications - Implements a RESTful interface for table and namespace operations following the Iceberg specification.
• Access Control - Provides a single interface to manage permissions across diverse databases, message queues, and object storage.
• Access Control Centralization - Provides centralized administrative control over permissions and security policies across multiple heterogeneous data sources.
• Centralized Permission Management - Centralizes identity validation and privilege enforcement via a single server before executing metadata operations.
• Access Management Policies - Retrieves and manages detailed policy rules associated with specific metadata assets to govern access.
• Data Governance Policies - Implements governance policies to control access and visibility for data collections and tables across a federated metadata lake.
• Metadata - Provides a framework for defining built-in or custom rules to govern catalogs, schemas, and AI models.
• Table-Level Access Controls - Grants and revokes granular permissions specifically for catalogs, schemas, and tables.
• Hierarchical Access Controls - Automatically applies access privileges from parent metalakes or catalogs down to all child metadata objects.
• API Request Authentication - Secures metadata access by verifying identities using OAuth 2.0, Basic Auth, and Kerberos.
• Role-Based Access Control - Implements a role-based access control system to manage user permissions via defined roles and groups.
• User Identity Management - Manages the lifecycle of user identities and groups to organize entities within the federated metadata system.
• Metadata Containers - Implements high-level metadata containers to group and organize diverse data sources.
• Model Schemas - Creates schemas within a model catalog to provide logical grouping for AI models.
• Multi-Protocol Tool Exposures - Runs a Model Context Protocol server to expose metadata as tools for AI agents.
• Data Lineage - Captures and processes the flow of data from source to destination using the OpenLineage standard.
• MCP Servers - Implements a Model Context Protocol server to expose metadata read operations and tag management to AI tools.
• Model Versioning Systems - Links and retrieves specific AI model versions using associated URIs and aliases.
• Hudi Catalog Integration - Lists Hudi schemas and partitioned tables via a catalog proxy for a unified view.
• Messaging Catalogs - Configures metadata catalogs for messaging systems like Kafka to enable unified access.
• AI Model Namespaces - Organizes machine learning model metadata using hierarchical namespaces and storage path mappings.
• User Activity Logs - Records system events and user actions through a configurable audit log framework with custom writers.
• Iceberg Table Lifecycle Management - Performs DDL operations to create and modify Iceberg schemas and tables across various storage backends.
• REST Server Hosting - Runs an Iceberg REST service as a standalone or embedded process to manage table metadata.
• Hive Metastore Services - Integrates with Hive Metastore to create and manage databases and tables for unified metadata governance.
• Metadata Unification - Provides a single point of access to metadata for diverse catalogs within a Trino environment.
• Data Partitioning - Organizes table data using identity or functional partitioning to improve storage efficiency.
• Data Partitioning Configuration - Implements range and list partitioning strategies to organize data based on column attributes.
• Storage Backend Configuration - Sets storage backend credentials and defines physical locations for lakehouse data.
• Data Type Mappings - Translates data types between a unified metadata format and specific table formats.
• Table Metadata Updates - Provides capabilities to update table metadata such as column types, nullability, and default values.
• Database Statistics Retrieval - Retrieves structural and performance statistics for metadata objects and data partitions to optimize query execution.
• Regional Replication - Replicates metadata across different geographical regions and cloud providers for distributed architectures.
• Sharding Strategies - Defines how data is distributed across a cluster using hash or random strategies.
• Global Metadata Synchronization - Connects instances across different regions to provide a global view of metadata.
• Lance REST Catalog Servers - Provides a standalone REST server to integrate Lance tables and dataset versioning.
• Metadata Management Interfaces - Provides a graphical interface for interacting with and overseeing the federated metadata environment.
• MySQL Schema Management - Performs DDL operations to create and delete MySQL databases, tables, and indexes.
• Table Structure Alterations - Supports structural alterations of tables, including renaming and modifying columns.
• OpenLineage Metadata Stores - Captures and stores data flow and lineage events using the OpenLineage standard across distributed engines.
• Physical Data Layouts - Defines data distribution and sort orders to optimize how tables organize data on disk.
• Query Plan Caching - Stores results of scan plans by snapshot ID to accelerate repeated queries with identical parameters.
• Authorization - Translates internal authorization models into data source-specific permissions for native access control.
• Model Metadata Retrieval - Retrieves detailed operational metadata and version aliases for specific AI models.
• Secure Access Tokens - Generates temporary access tokens to securely delegate access to private cloud storage resources like S3 and GCS.
• SQL Execution Interfaces - Performs standard data definition and manipulation operations across federated data assets using SQL.
• Index Optimizations - Creates primary key indexes and joint primary keys to optimize data retrieval performance.
• Table Definitions - Provides primitives for defining database table structures and column data types using CSV definitions.
• Column Structural Modifications - Updates table structures by adding, deleting, renaming, or reordering columns.
• Tag Definitions - Provides tools to create, update, and delete the definitions of tags used for metadata categorization.
• DDL Executions - Executes DDL operations such as table renaming and column type adjustments across federated sources.
• Metadata Management CLIs - Creates, updates, and deletes metadata entities and properties through a command line interface.
• Metadata Synchronizers - Implements automated synchronization to ensure metadata changes are reflected immediately in the underlying data sources.
• Container Deployment Configurations - Ships the core metadata service using container images with configurable memory and catalog settings.
• Storage Cluster Orchestration - Assigns schemas to different clusters using independent storage locations within a single catalog.
• Messaging Topic Governance - Creates and deletes messaging topics within a schema to govern streaming assets.
• Topic Registration - Registers topics within messaging catalogs to organize streaming data assets.
• Event-Driven Lineage - Processes real-time lineage metadata through a server-side framework to update external systems and dependency graphs.
• Metadata Asset Auditing - Allows retrieval of metadata objects associated with specific tags and listing all tags applied to an object.
• Object Tagging - Assigns descriptive tags to metadata assets with support for inheritance from parent objects.
• Authentication Extensions - Extends server authenticator interfaces and client token providers to support proprietary identity logic.
• Cloud Credential Management - Automatically manages cloud storage authentication to eliminate manual credential configuration.
• Data Access Policies - Links security policies directly to metadata objects to govern data access and regulatory compliance.
• External Identity Provider Integration - Connects to OAuth 2.0 and OIDC providers using signing keys or JWKS endpoints.
• Identity Authentication - Verifies identities using environment variables, OAuth 2.0 JWT tokens, or Kerberos tickets.
• Interface Access Security - Secures the metadata management interface and data access layers using HTTPS, CORS, and identity providers.
• Identity Mapping - Transforms external provider principals into internal user identities using regex-based mapping rules.
• Table Maintenance Automations - Executes maintenance jobs based on table statistics and predefined rules using templates.
• Authentication Overrides - Enables setting authentication types at the catalog, schema, or fileset level using a hierarchical override system.
• Model Version Histories - Maintains version history for AI models using unique numbers or aliases to manage their lifecycle.
• Administrative Interfaces - Provides a command-line interface and web dashboard for administrative metadata operations.
• Kafka Metadata Management - Creates and updates metadata for Apache Kafka topics across clusters from a unified interface.
• Database User Groups - Organizes users into groups to simplify the assignment of roles and privileges across the metadata lake.
• Data Catalogs - Unified metadata lake for managing data and AI assets.
• Data Lake Management - Unified metadata management for data lakes and warehouses.
• Open Source Catalogs - High-performance, unified metadata lake for data and AI assets.

سجل النجوم