Awesome GitHub RepositoriesDocument Chunking and Embedding Pipelines

QAnything is a retrieval-augmented generation application framework and self-hosted AI interface. It functions as a system that combines a vector database knowledge base, a document parsing service, and a hybrid search engine to generate answers based on private user data. The project features a modular pipeline architecture that allows users to independently replace components such as parsers, embedding models, and reranking engines. It supports local-first model deployment and offline operation to ensure data privacy, and includes a two-stage retrieval pipeline that merges dense vector embe

Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management. The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer

pgai 是一个 PostgreSQL AI 工具包和框架，旨在将大语言模型和向量嵌入直接集成到数据库中。它充当了在标准数据库查询中执行机器学习模型请求和进行文本转 SQL 翻译的桥梁。 该项目提供了一个自动化的向量嵌入流水线，负责处理来自表和非结构化文档的文本加载、解析和分块。该系统利用后台工作进程在源数据发生变化时自动同步嵌入，并包含用于构建检索增强生成（RAG）应用和语义搜索引擎的专用工具。 该工具包涵盖了广泛的功能领域，包括利用 OCR 处理非结构化数据、创建将数据库模式映射到自然语言的语义目录，以及通过向量索引和结果重排序实现高性能相似度搜索。它还支持通过 SQL 调用外部模型，从而实现数据增强、分类和内容审核。

该项目是一个容器化的开发栈和应用程序框架，用于构建检索增强生成（RAG）系统。它提供了一个 Docker 化的 AI 沙盒，集成了本地模型运行时、知识图谱和向量存储，以实现上下文感知聊天机器人的创建。 该栈的特点在于其基于图的向量存储，它结合了结构化知识图谱与向量索引，用于语义和结构化数据检索。它允许在 CPU 或 GPU 加速下进行本地模型托管，从而无需依赖外部云 API 即可执行生成任务。 该框架涵盖了广泛的功能，包括 PDF 文档的处理和索引、基于容器的 AI 服务编排，以及实现基于事实的响应生成。它包含一个带有增量响应流的 Web 聊天界面，以及用于在不同语言模型提供商之间切换的标准接口。 该环境通过容器编排进行引导，以快速部署预配置的模型和数据库栈。

Chonkie 是一个专为检索增强生成 (RAG) 流水线设计的文本分块库。它充当语义文本分割器和 RAG 数据摄取流水线，将原始文本转换为嵌入片段，以便存储在向量数据库中。 该项目通过专门的分割策略脱颖而出，包括用于保留源代码逻辑边界的基于 AST 的代码分割器，以及使用嵌入模型根据语义确定边界的语义文本分割器。它还提供了一个向量数据库摄取器，用于自动化生成嵌入并将其导出到各种存储中。 该库涵盖了广泛的功能，包括通过 OCR 和 Markdown 提取进行文档解析，多种分割方法（如基于 Token 计数和分层分割），以及通过可重用流水线进行工作流编排。它支持多种向量存储集成，包括 Qdrant、Milvus、Weaviate 和 Elasticsearch，以及将数据导出为 JSON 和 Hugging Face 数据集。 用户可以通过命令行界面执行这些操作，或将系统部署为容器化的 API 服务。

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind

Vector Admin is a multi-database administration platform and management interface designed for handling vector databases from a single dashboard. It provides a unified web interface that connects to multiple distinct vector database providers simultaneously, allowing administrators to oversee connections, manage user permissions through role-based access control, and enforce multi-tenant data access boundaries. The platform includes data management utilities for document ingestion, processing uploaded files through embedding generation pipelines, and inspecting or editing individual text chun