Awesome GitHub RepositoriesAsynchronous Processing

Chonkie 是一个专为检索增强生成 (RAG) 流水线设计的文本分块库。它充当语义文本分割器和 RAG 数据摄取流水线，将原始文本转换为嵌入片段，以便存储在向量数据库中。 该项目通过专门的分割策略脱颖而出，包括用于保留源代码逻辑边界的基于 AST 的代码分割器，以及使用嵌入模型根据语义确定边界的语义文本分割器。它还提供了一个向量数据库摄取器，用于自动化生成嵌入并将其导出到各种存储中。 该库涵盖了广泛的功能，包括通过 OCR 和 Markdown 提取进行文档解析，多种分割方法（如基于 Token 计数和分层分割），以及通过可重用流水线进行工作流编排。它支持多种向量存储集成，包括 Qdrant、Milvus、Weaviate 和 Elasticsearch，以及将数据导出为 JSON 和 Hugging Face 数据集。 用户可以通过命令行界面执行这些操作，或将系统部署为容器化的 API 服务。

This project is a web-based platform designed for retrieval-augmented generation, providing a conversational interface that connects language models to private document collections. It functions as a comprehensive system for managing enterprise knowledge bases, allowing users to query internal documents and receive context-aware, source-verified answers through a natural language chat interface. The platform distinguishes itself by integrating vector-based semantic search with modular knowledge base management, enabling the ingestion, segmentation, and indexing of documents into searchable em

This platform is a self-hosted knowledge management system designed for interacting with documents through natural language. It functions as a retrieval-augmented generation engine, allowing users to upload files and query them using large language models. The system provides a unified interface for document-based chat, ensuring that responses are grounded in the source material through specific citations. The platform distinguishes itself through a multi-tenant architecture that enforces strict data isolation between users and organizations. It features a flexible AI gateway that standardize