Awesome GitHub RepositoriesPluggable Storage Backends

Mempalace is a local-first long-term memory store for large language models and AI agents. It provides a persistent storage system for verbatim conversation history and agent data, utilizing a local-first knowledge graph to track evolving entity relationships and timelines. The project implements a standardized memory protocol that allows external AI clients to read and write persistent memory via standard input and output. It features a hybrid semantic search engine that combines keyword boosting and reranking to find precise historical information across scoped categories. The system inclu

CodiMD is a real-time collaborative markdown editor and self-hosted knowledge base. It provides a shared workspace where multiple users can write and format notes using markdown syntax simultaneously. The platform extends basic editing by transforming markdown content into interactive mind maps for data visualization and structured visual slides for presentations. It functions as an OAuth compatible note server, integrating with external identity providers to manage user authentication and access control. The system supports self-hosting and cloud platform deployment, allowing users to maint

elasticsearch-dump is a command-line tool for importing and exporting data, mappings, and settings between Elasticsearch and OpenSearch clusters. It serves as a backup and restore system, a data transfer utility for cross-cluster migrations, and a bridge for streaming search index data to and from S3-compatible storage. The tool functions as a JSON index data transformer, allowing the use of custom JavaScript modules to modify or anonymize documents in flight during the migration process. It also enables the creation of search bodies using external JavaScript templates to customize the data b

Noms is a distributed version control database and content-addressable data store. It identifies data by cryptographic hashes to ensure integrity and deduplication, while tracking dataset state changes through a sequence of immutable commits to enable branching, forking, and historical recovery. The system functions as a peer-to-peer data synchronizer, reconciling state between disconnected database instances to ensure all nodes converge on the same data. It distinguishes itself as a schema-flexible document store that supports self-describing types, allowing schemas to evolve and widen as ne

Alluxio is a virtual distributed file system and data orchestration layer that serves as a high-performance caching layer between cloud storage and compute clusters. It acts as a distributed data cache designed to accelerate data access for large-scale analytics and machine learning workloads. The system provides a unified interface that presents multiple heterogeneous storage backends as a single coherent namespace. This allows for the unification of diverse storage systems, enabling computation engines to access data from different providers without changing application code. The project c

gopass is a terminal-based password manager and GPG secret store used for generating, storing, and retrieving encrypted credentials. It functions as a collaborative secret manager that encrypts data using GPG or age and synchronizes it across devices and teams using Git. The system distinguishes itself by treating version control repositories as the primary storage backend, enabling secure secret sharing and version history for credentials. It utilizes a hierarchical directory structure to organize secrets on the filesystem and supports multi-store mounting to combine multiple independent rep

SpiceDB is a distributed permission store and relationship-based access control system. It provides a scalable database for storing and querying fine-grained authorization relationships, implementing a consistency model inspired by Google Zanzibar to manage access rights across large-scale applications. The system uses a dedicated schema language to define the rules and logic governing how relationships translate into permissions independently of application code. It functions as a pluggable authorization engine that persists relationship tuples in external relational databases such as Postgr

This project is a multimodal AI proxy and content generation hub that provides a unified web interface for interacting with multiple large language models and generative AI services. It functions as a secure API access gateway, routing requests from a single dashboard to various external AI backends using configurable base URLs and API keys. The platform is delivered as a cross-platform progressive web application, allowing for installation on Linux, Windows, and MacOS. It distinguishes itself by consolidating text, image, audio, and video generative controls into a standardized interface, su

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Apache Zeppelin is a web-based notebook platform for interactive data analytics that supports executing code in over 20 languages within a single notebook. It provides a plugin-based interpreter architecture that allows the notebook to be extended with new languages and data sources, and includes a JDBC connector abstraction for connecting to any JDBC-compliant database. The platform also features session-isolated interpreter contexts, enabling separate interpreter instances per notebook or user with support for dependency injection and user impersonation. The platform distinguishes itself th

Cronicle is a distributed job scheduler that replaces traditional cron with a browser-based management interface. It runs scheduled tasks across a cluster of servers with automatic failover, using a custom cron parser that intersects day-of-month and day-of-week constraints when both are specified. The system executes jobs through a plugin framework that runs command-line scripts in any language, communicating via JSON over standard input and output. The scheduler provides a web-based real-time dashboard for monitoring running jobs with live logs, resource usage charts, and progress updates.

Certmagic 是一个用于自动化 TLS 证书颁发和续期的 Go 库。它作为一个自动 HTTPS 配置器和 ACME 客户端，处理证书的整个生命周期，以确保在无需人工干预的情况下实现安全连接。 该库的特色在于支持按需 TLS 配置，它根据服务器名称在 TLS 握手期间动态生成证书。它还通过 DNS 挑战验证为通配符证书提供自动化，并与 ZeroSSL API 集成以获取证书。 该项目涵盖了广泛的功能，包括 ACME 挑战解析、自动续期调度和证书吊销。它通过可插拔的存储后端管理数据持久化，并实现分布式锁和同步以协调服务器集群间的证书管理。其他功能包括 OCSP 装订、内存缓存以及为自定义服务器生成 TLS 配置。

dlt 是一个 Python 数据摄取工具和 ETL 流水线框架，旨在从不同来源获取数据并将其持久化到结构化目标中。它作为一个模式推断引擎，可自动检测数据类型并将嵌套的 JSON 结构扁平化为关系表，将数据从源端移动到数据湖、数据仓库或向量数据库。 该项目通过 AI 驱动的流水线生成脱颖而出，利用大语言模型为 REST API 构建提取代码和连接器。它还支持多模态向量存储和向量数据库的专门填充，以支持 AI 和机器学习应用。 该框架涵盖了广泛的功能，包括自动化模式演进、通过状态跟踪进行增量数据加载，以及通过强制执行数据契约进行数据质量验证。它提供了用于关系数据规范化、加载前后转换的工具，以及针对 SQL 数据库和云对象存储的多种目标适配器。 可观测性通过流水线执行仪表板、列血缘跟踪以及使用基于内容的哈希进行模式版本验证来处理。

Waterline 是一个基于适配器的 Node.js 数据库库和对象关系映射器（ORM）。它作为一个多数据库持久化层，将标准方法调用转换为针对 SQL 和 NoSQL 数据库的特定查询。 该库为管理不同存储（包括 MySQL、PostgreSQL、MongoDB 和 Microsoft SQL Server）提供了统一的接口。这种架构允许在单个应用中集成多个数据库后端，并通过更换适配器而非修改业务逻辑来简化数据库迁移。 该项目涵盖了关系型和文档型映射，为数据存储和检索提供了统一的语法。它利用模式驱动的对象映射来弥合数据库表或集合与 JavaScript 对象之间的鸿沟。

本项目是官方 Kubernetes 文档网站，作为管理容器化应用的综合技术资源。它作为一个开源技术文档门户，为分布式系统软件提供指南、教程和参考资料。 该网站使用带有组件化模板架构的静态网站生成器构建，以保持一致的设计模式。它具有一个 OpenAPI 文档生成器，可解析技术规范以自动构建和更新结构化的 API 参考页面。为了支持全球受众，它采用国际化感知的内容路由来管理手册的本地化版本。 开发工作流包括用于预览站点更改的热重载服务器和目标语言渲染，以加快构建时间。该项目涵盖了广泛的技术领域，包括集群编排、网络配置和资源管理。

ezbookkeeping 是一个自托管的个人财务管理工具，旨在跟踪收入和支出。它作为一个多币种记账工具，通过私有应用程序管理账户余额和总资产。 该系统的特色在于集成了语言模型交易处理器，可从收据图像中提取数据并自动录入财务条目。它还提供基于 REST 的财务 API 和相应的命令行客户端，支持程序化财务自动化以及将记录与外部脚本集成。 该平台包含一个财务数据分析仪表板，用于生成支出趋势图和比例图。它通过自定义类别和标签、具有自动汇率同步功能的多币种资产管理以及具有多因素身份验证的安全访问控制，实现了全面的数据管理。 数据可以持久化到本地目录或远程云存储，用户界面通过专门的翻译文件和区域格式配置支持本地化。

Incus is a unified orchestration platform for managing system containers, OCI application containers, and virtual machines through a single control plane. It brings together cluster infrastructure management, secure multi-tenancy, software-defined networking, and pluggable storage backend orchestration into one cohesive system exposed via a full REST API and command-line interface. What distinguishes Incus is its ability to run multiple instance types side by side—full Linux system containers, OCI application containers, and QEMU virtual machines—all managed with consistent tooling. Networkin

Dkron is a distributed, fault-tolerant system designed for scheduling and executing recurring tasks across a cluster of nodes. It functions as a cron-based orchestrator that manages job lifecycles, including automatic retries, timeouts, and complex dependencies, while ensuring state consistency through a consensus protocol. By coordinating remote task execution across infrastructure, it enables the automation of background operations and the management of distributed workflows. The system distinguishes itself through a modular architecture that supports pluggable storage backends and a plugin

myDrive 是一个自托管的云存储服务器和基于 Web 的文件管理器。它作为一个加密文件存储系统，允许用户在自己的硬件上托管和管理个人文档及媒体。 该系统充当多后端存储网关，能够将文件数据路由到包括本地磁盘或远程云存储桶在内的各种位置。它使用文档数据库来管理文件元数据、文件夹结构和用户组织。 该平台提供文件上传和下载、通过回收站系统组织内容，以及通过安全链接共享私有文件的功能。它包含一个用于远程媒体管理的 Web 界面，内置缩略图和媒体预览。访问通过基于电子邮件的身份验证和行业标准数据加密进行保护。

Hawk 是一个适用于 Android 的安全本地存储库和持久化 API。它提供了一个框架，用于使用键值存储系统保存和检索原始类型、对象和集合。 该库具有加密存储包装器，通过可插拔的加密提供程序保护数据，包括回退到非加密模式的机制。其架构允许自定义存储后端实现和基于接口的组件注入，以替换默认的序列化、解析和转换过程。 该系统支持类型保留序列化，并包含用于管理数据生命周期的实用程序，例如目标条目删除和批量状态重置。它还结合了系统日志拦截功能，以捕获和处理内部消息。