Awesome GitHub RepositoriesCloud Storage Definition Loading

DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data

Elsa Core is a workflow engine framework designed for defining, executing, and managing long-running business processes. It functions as a distributed workflow orchestrator and event-driven trigger system, capable of operating as a multi-tenant platform with secure data isolation. The project distinguishes itself through a flexible approach to workflow definitions, supporting a visual drag-and-drop designer, programmatic C# definitions, and portable JSON specifications. It provides a highly extensible architecture allowing for the development of custom activities and the use of a dynamic expr

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

pgai 是一个 PostgreSQL AI 工具包和框架，旨在将大语言模型和向量嵌入直接集成到数据库中。它充当了在标准数据库查询中执行机器学习模型请求和进行文本转 SQL 翻译的桥梁。 该项目提供了一个自动化的向量嵌入流水线，负责处理来自表和非结构化文档的文本加载、解析和分块。该系统利用后台工作进程在源数据发生变化时自动同步嵌入，并包含用于构建检索增强生成（RAG）应用和语义搜索引擎的专用工具。 该工具包涵盖了广泛的功能领域，包括利用 OCR 处理非结构化数据、创建将数据库模式映射到自然语言的语义目录，以及通过向量索引和结果重排序实现高性能相似度搜索。它还支持通过 SQL 调用外部模型，从而实现数据增强、分类和内容审核。

AliSQL is a fork of MySQL by Alibaba that extends the relational database management system with enhancements for high performance, scalability, and enterprise-grade availability. It retains the core MySQL identity as a SQL-based database for storing, organizing, and retrieving structured data, while adding optimizations for large-scale transactional and analytical workloads. The project differentiates itself through a set of Alibaba-specific improvements, including a columnar engine for accelerating analytical queries directly on MySQL tables, and a distributed, shared-nothing NDB Cluster en

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

lakeFS 是一个数据湖版本控制系统，为存储在对象存储中的大型数据集提供类似 Git 的分支和提交功能。它作为一个版本控制层，支持创建不可变快照、原子提交和零拷贝分支，从而在不复制物理文件的情况下为数据实验创建隔离环境。 该系统充当 S3 兼容的存储网关和 Iceberg REST 目录，允许标准云存储协议和兼容客户端管理版本化表。它通过使用事件驱动的钩子系统在更改合并到生产环境之前根据治理策略验证数据集，从而充当数据质量守门人。 该平台涵盖了广泛的数据治理功能，包括 Pull Request 协作、基于角色的访问控制和数据血缘追踪。它为工作流编排、机器学习管线和各种大数据计算引擎提供了集成，支持多云存储连接以及通过 SSO 和 SCIM 进行身份同步。 该软件可以使用二进制文件、容器或 Helm Chart 安装，以便在 Kubernetes 上部署。

GAM is a command-line tool for administering Google Workspace and Cloud Identity. It translates command-line arguments into structured API calls, enabling administrators to manage users, groups, organizational units, and domain settings across a Google Workspace environment. The tool handles authentication through OAuth2 flows, service accounts, and workload identity federation, and supports multi-tenant configurations for managing multiple domains or cloud projects from a single installation. GAM distinguishes itself through its batch processing and automation capabilities. It can process la

该项目是一个 AWS pandas 集成库和数据流水线框架，旨在简化本地内存与 AWS 存储及分析服务之间的数据移动和转换。它作为一个云数据湖工具包和存储文件管理器，允许用户在各种云环境中读取、写入和转换结构化数据。 该库作为分布式计算编排器脱颖而出，能够在 EMR 等环境中管理集群，以处理超出单机内存限制的数据集。它还提供用于管理向量索引和在云存储桶内执行相似度搜索的专门功能。 其更广泛的功能面涵盖了针对 DynamoDB、RDS 和 Timestream 等服务的云数据库 ETL，以及通过 AWS Glue 进行的云数据目录管理。它支持通过 Athena 和 Redshift 进行无服务器数据分析，并提供用于管理 S3 对象、在 OpenSearch 中索引文档以及分析 CloudWatch 日志的实用程序。

aws-sdk-pandas 是一个 Python 库，将 pandas 数据帧与 AWS 服务集成，充当云数据 ETL 工具和数据湖连接器。它提供了一个统一界面，用于在内存中数据帧与云存储、数据库和数据仓库之间移动和转换数据。 该项目作为分布式计算编排器脱颖而出，能够将基于 pandas 的工作负载提交到 EMR 集群和无服务器处理环境。它进一步专门通过 Ray 集群初始化来协调分布式数据处理，以处理超出单机内存的数据集。 该库涵盖了广泛的功能，包括 S3 的对象存储管理、Athena 和 Redshift 的 SQL 查询执行，以及与 NoSQL、图和时间序列数据库的集成。它还包括通过 Glue 目录进行元数据管理、OpenSearch 数据索引以及在 QuickSight 中管理商业智能资产的实用程序。 其他功能包括检索密钥、分析 CloudWatch 日志以及管理数据质量规则集。