Awesome GitHub RepositoriesVector Document Indexing

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

PrivateGPT is a private AI document assistant and local knowledge base manager designed for querying private files and documents using retrieval-augmented generation. It functions as a local language model application and API gateway, allowing users to obtain cited answers from unstructured data without sending information to external servers. The system differentiates itself by acting as a tool integrator that connects language models to external functions, including web search, tabular data analysis, and custom action extensions. It provides a standardized API layer that allows local infere

oh-my-codex is an AI coding workflow orchestrator and a retrieval augmented generation documentation assistant. It manages complex programming tasks through a structured sequence of planning, execution, and verification phases, while providing tools for querying and translating technical documentation. The project utilizes Git worktrees to isolate parallel coding sessions, ensuring that concurrent tasks remain independent. It integrates a vector-store knowledge base to index documents into embeddings, enabling semantic search and factual context retrieval across multiple languages. The syste

Gitbook is a documentation-as-code platform designed for centralized technical knowledge management. It functions as a knowledge management system that synchronizes documentation files directly with version control repositories, allowing teams to maintain content alongside their source code. The platform distinguishes itself through an integrated artificial intelligence layer that provides context-aware search assistance and automated content suggestions. By utilizing block-based content modeling, it enables the construction of structured, modular documentation that can be compiled into stati

Kotaemon is an orchestration framework designed for building modular, agentic workflows that integrate document processing, retrieval-augmented generation, and multi-step reasoning. It provides a comprehensive platform for developing document-based question answering systems, allowing users to chain language models, prompt templates, and external tools into complex, automated pipelines. The system distinguishes itself through a highly modular architecture that emphasizes component-based composition and schema-driven data exchange. It supports autonomous agents capable of decomposing complex q

Mastra is an orchestration framework designed for building, deploying, and managing autonomous AI agents and multi-agent systems. It provides a comprehensive suite of primitives for creating resilient AI applications, including durable workflow orchestration, event-driven agent loops, and semantic memory management. By integrating these core components, the platform enables developers to build complex, multi-step processes that can reason about goals and execute tasks without manual intervention. The framework distinguishes itself through its focus on observability and secure, isolated execut

This project is a retrieval-augmented generation pipeline designed for building custom ChatGPT plugins that allow language models to query private or professional documents. It implements a full retrieval workflow, from processing and indexing document chunks to retrieving relevant context for natural language queries. The system distinguishes itself through a hybrid retrieval approach that combines dense vector embeddings with sparse keyword matching, further refined by a two-stage semantic re-ranking process. It includes specialized data privacy tools for screening personally identifiable i

This repository is a collection of frameworks and guides for Llama models, functioning as a fine-tuning framework, an inference pipeline, and an AI workflow orchestrator. It provides tools for adapting large language models to specific datasets and domains. The project includes a parameter-efficient fine-tuning toolkit that utilizes techniques like low-rank adaptation to reduce memory and compute requirements. It also serves as an implementation guide for retrieval-augmented generation, combining model inference with external data retrieval to improve response accuracy. The capability surfac

DocsGPT is a retrieval-augmented generation platform and private knowledge base used to build AI agents that perform grounded search and analysis. It functions as a multi-model AI orchestrator and enterprise agent builder, allowing for the integration of various local and cloud language models to customize reasoning and text generation. The project provides a visual environment for developing automated assistants using conditional logic and third-party API connectivity. It enables the creation of private AI agents capable of performing enterprise search and detailed document analysis using pr

SurfSense is a self-hosted platform designed for building retrieval-augmented generation pipelines and managing private knowledge bases. It functions as a containerized research stack that allows users to index diverse data sources and query them using language models, ensuring that all information retrieval is grounded in specific source citations. The platform distinguishes itself through its modular architecture, which supports the integration of custom tools and diverse language models via a unified abstraction layer. It facilitates secure, collaborative research environments by implement

QAnything is a retrieval-augmented generation application framework and self-hosted AI interface. It functions as a system that combines a vector database knowledge base, a document parsing service, and a hybrid search engine to generate answers based on private user data. The project features a modular pipeline architecture that allows users to independently replace components such as parsers, embedding models, and reranking engines. It supports local-first model deployment and offline operation to ensure data privacy, and includes a two-stage retrieval pipeline that merges dense vector embe

Unstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t

Eino is an AI agent development kit and LLM application framework designed for building autonomous agents and orchestrating complex language model workflows. It serves as a multi-agent orchestration engine and workflow orchestrator, providing a graph-based execution model to route data between models, tools, and retrievers. The framework distinguishes itself through a robust set of multi-agent coordination patterns, including supervisor-led management, sequential flows, and autonomous reasoning loops like ReAct. It features advanced agent execution controls such as active turn preemption, che

Genkit is an LLM application framework and generative AI developer toolkit designed for building production AI applications. It serves as an AI workflow orchestrator that coordinates model calls and agentic tool usage through type-safe execution flows. The project provides a unified model interface and plugin architecture to standardize access to diverse large language models, vector stores, and telemetry backends. It distinguishes itself with a dedicated observability suite for tracing execution steps and a developer toolkit for prompting, debugging, and evaluating AI logic via a local inter

Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management. The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer

pgai is a PostgreSQL AI toolkit and framework designed to integrate large language models and vector embeddings directly into a database. It serves as a bridge for executing machine learning model requests and performing text-to-SQL translations within standard database queries. The project provides an automated vector embedding pipeline that handles the loading, parsing, and chunking of text from tables and unstructured documents. This system utilizes a background worker to synchronize embeddings automatically as source data changes and includes specialized tools for building retrieval-augme

This project is a containerized development stack and application framework for building retrieval-augmented generation systems. It provides a dockerized AI sandbox that integrates local model runtimes, knowledge graphs, and vector stores to enable the creation of contextual chatbots. The stack is distinguished by its graph-based vector store, which combines structured knowledge graphs with vector indices for both semantic and structural data retrieval. It allows for local model hosting with CPU or GPU acceleration, enabling generative tasks without reliance on external cloud APIs. The frame

यह प्रोजेक्ट एक नॉलेज बेस प्लगइन और RAG कॉन्टेक्स्ट मैनेजर है जो सिमेंटिक सर्च और रिलेशनशिप मैपिंग को सक्षम करने के लिए एक स्थानीय वेक्टर डेटाबेस इंटरफेस का उपयोग करता है। यह कीवर्ड मिलान के बजाय वैचारिक अर्थ के आधार पर सिमेंटिक रूप से संबंधित नोट्स और अंशों को खोजने के लिए टेक्स्ट को संख्यात्मक वैक्टर में बदलता है। यह सिस्टम एक सिमेंटिक ग्राफ़ विज़ुअलाइज़र के माध्यम से खुद को अलग करता है जो वैचारिक कनेक्शन को प्रकट करने के लिए नोट्स को क्लस्टर में मैप करता है। इसमें एक कॉन्टेक्स्ट मैनेजर भी है जो बड़े भाषा मॉडल वार्तालापों के लिए आधारभूत तथ्यात्मक आधार प्रदान करने के लिए स्थानीय नोट्स और अंशों को पुन: प्रयोज्य पैक में बंडल करने में सक्षम है। यह टूल प्राकृतिक भाषा ज्ञान क्वेरी, नोट निर्माण के लिए स्वचालित वर्कफ़्लो निष्पादन, और स्थानीय तथा क्लाउड-आधारित AI मॉडल के बीच प्रॉम्प्ट को रूट करने की क्षमता सहित क्षमताओं की एक विस्तृत श्रृंखला को कवर करता है। यह संपादन प्रक्रिया के दौरान समान दस्तावेज़ों को सतह पर लाने के लिए इनलाइन संबंधित सामग्री संकेतक और एक फ़ूटर पैनल जैसे कई खोज इंटरफेस प्रदान करता है।

zvec is an embedded vector database engine and indexing library designed for high-dimensional similarity search. It functions as a hybrid search engine and a retrieval-augmented generation knowledge base, allowing for the storage and retrieval of dense and sparse vectors. The system is distinguished by its hybrid retrieval pipeline, which fuses vector similarity, full-text keyword matching, and scalar metadata filtering into single query operations. It supports a plugin-based model integration system for registering custom embedding models and rerankers, as well as language bindings for nativ

DrQA एक ओपन-डोमेन प्रश्न उत्तर प्रणाली है जो एक बड़े कॉर्पस से प्रासंगिक दस्तावेज़ों को पुनः प्राप्त करती है और प्राकृतिक भाषा के प्रश्नों के विशिष्ट उत्तर निकालती है। इसे एक न्यूरल नेटवर्क सिस्टम के रूप में लागू किया गया है जो मशीन रीडिंग कॉम्प्रिहेंशन मॉडल के साथ एक दस्तावेज़ रिट्रीवल इंजन को जोड़ता है। यह सिस्टम टू-स्टेज पाइपलाइन आर्किटेक्चर का उपयोग करता है। एक कोर्स-ग्रेन्ड दस्तावेज़ रिट्रीवर संभावित दस्तावेज़ों की पहचान करने के लिए वेटेड वर्ड वेक्टर्स का उपयोग करता है, जबकि एक फाइन-ग्रेन्ड मशीन रीडिंग कॉम्प्रिहेंशन मॉडल उत्तर वाले सटीक टेक्स्ट स्पैन की पहचान करता है और उसे निकालता है। इस प्रोजेक्ट में एक सुपरवाइज्ड नेचुरल लैंग्वेज प्रोसेसिंग डेटासेट जनरेटर भी शामिल है। यह टूल स्वचालित स्ट्रिंग ह्यूरिस्टिक्स का उपयोग करके प्रश्न-उत्तर जोड़ियों को सहायक पैराग्राफ के साथ मिलान करके ट्रेनिंग उदाहरण बनाता है। कोडबेस न्यूरल नेटवर्क प्रोसेसिंग के लिए रॉ टेक्स्ट तैयार करने के लिए टेक्स्ट प्रोसेसिंग और टोकनाइज़ेशन के लिए अतिरिक्त क्षमताएं प्रदान करता है।