# Retrieval-Augmented Generation Frameworks

> Search results for `build a retrieval-augmented generation pipeline over your documents` on awesome-repositories.com. 117 total matches; showing the first 50.

Explore on the web: https://awesome-repositories.com/q/build-a-retrieval-augmented-generation-pipeline-over-your-documents

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [this search on awesome-repositories.com](https://awesome-repositories.com/q/build-a-retrieval-augmented-generation-pipeline-over-your-documents).**

## Results

- [openai/chatgpt-retrieval-plugin](https://awesome-repositories.com/repository/openai-chatgpt-retrieval-plugin.md) (21,192 ⭐) — This project is a retrieval-augmented generation pipeline designed for building custom ChatGPT plugins that allow language models to query private or professional documents. It implements a full retrieval workflow, from processing and indexing document chunks to retrieving relevant context for natural language queries.

The system distinguishes itself through a hybrid retrieval approach that combines dense vector embeddings with sparse keyword matching, further refined by a two-stage semantic re-ranking process. It includes specialized data privacy tools for screening personally identifiable i
- [infiniflow/ragflow](https://awesome-repositories.com/repository/infiniflow-ragflow.md) (82,922 ⭐) — This project is a comprehensive retrieval-augmented generation platform designed for building, managing, and deploying knowledge-based AI applications. It provides a unified environment for organizing datasets, configuring conversational chat assistants, and developing autonomous agents that execute multi-step reasoning workflows. By integrating document intelligence with advanced retrieval pipelines, the platform enables the creation of grounded, verifiable responses supported by traceable citations.

The platform distinguishes itself through deep document understanding and sophisticated know
- [codecrafters-io/build-your-own-x](https://awesome-repositories.com/repository/codecrafters-io-build-your-own-x.md) (516,240 ⭐) — This project provides a comprehensive framework for creating, managing, and executing educational programming challenges. It includes standardized systems for authoring instructional content, defining test cases, and structuring documentation to ensure consistent learning outcomes. The platform supports a wide range of programming languages through dedicated execution environments that handle compilation, dependency management, and automated testing.

The infrastructure facilitates both local and remote development workflows, offering command-line utilities for testing code without requiring v
- [griptape-ai/griptape](https://awesome-repositories.com/repository/griptape-ai-griptape.md) (2,541 ⭐) — Griptape is a Python framework for building generative AI applications, autonomous agents, and complex AI workflows. It functions as both an AI agent orchestrator and a workflow engine, capable of managing sequential pipelines and directed acyclic graphs to ensure predictable execution of AI tasks.

The framework distinguishes itself through a focus on security and governance, utilizing a Docker-based environment to execute model-generated code and shell commands in isolation. It employs a driver-based abstraction layer that allows developers to swap language model providers and vector stores
- [zylon-ai/private-gpt](https://awesome-repositories.com/repository/zylon-ai-private-gpt.md) (57,278 ⭐) — This project is a privacy-first backend service designed to facilitate retrieval-augmented generation by processing local documents into searchable vector representations. It provides a modular architecture that allows users to ingest diverse file formats, manage document metadata, and perform semantic searches to provide context-aware responses for chat and completion requests.

The system distinguishes itself through a database-agnostic abstraction layer that supports various storage backends, ranging from local disk storage to enterprise-grade vector databases. It offers flexible deployment
- [emma1066/retrieval-augmented-it-openner](https://awesome-repositories.com/repository/emma1066-retrieval-augmented-it-openner.md) (27 ⭐) — This is the github repository for the paper: Retrieval Augmented Instruction Tuning for Open NER with Large Language Models.
- [buildthingsuseful/build-your-own-kafka](https://awesome-repositories.com/repository/buildthingsuseful-build-your-own-kafka.md) (65 ⭐) — Build Your Own Kafka
- [bragai/brag-langchain](https://awesome-repositories.com/repository/bragai-brag-langchain.md) (4,028 ⭐) — bRAG-langchain is a framework for building retrieval augmented generation pipelines using LangChain to connect documents with language models. It functions as a vector store orchestrator that manages document indexing and retrieval strategies to improve context accuracy.

The system implements an advanced retrieval pipeline featuring a semantic query router that directs natural language inputs to specific data sources or prompts. It includes a metadata filtering engine that translates natural language queries into structured schemas to narrow search results.

The project covers hybrid search o
- [deepset-ai/haystack](https://awesome-repositories.com/repository/deepset-ai-haystack.md) (24,253 ⭐) — Haystack is an orchestration framework designed for building complex search and generative AI pipelines. It functions as an agentic workflow engine, enabling the construction of automated sequences that allow AI agents to perform multi-step reasoning and data analysis.

The framework utilizes a modular, component-based architecture that connects processing steps into directed acyclic graphs. By employing a provider-agnostic integration layer, it decouples core logic from specific external AI services and vector databases, allowing for the flexible exchange of underlying technologies. This desi
- [aishwaryanr/awesome-generative-ai-guide](https://awesome-repositories.com/repository/aishwaryanr-awesome-generative-ai-guide.md) (24,755 ⭐) — This project is a community-driven knowledge repository and technical learning resource focused on the field of generative artificial intelligence. It serves as a centralized hub for developers and practitioners to access curated research, tutorials, and foundational concepts necessary for building and deploying modern artificial intelligence applications.

The platform distinguishes itself through a collaborative, distributed contribution model that aggregates diverse learning materials into a structured, searchable knowledge base. It covers a wide range of specialized topics, including retri
- [abetlen/llama-cpp-python](https://awesome-repositories.com/repository/abetlen-llama-cpp-python.md) (9,993 ⭐) — llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs.

The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
- [peiyuanix/build-your-own-zerotier](https://awesome-repositories.com/repository/peiyuanix-build-your-own-zerotier.md) (603 ⭐) — Build your own layer-2 virtual switch in less than 300 lines of code
- [anthropics/claude-cookbooks](https://awesome-repositories.com/repository/anthropics-claude-cookbooks.md) (45,835 ⭐) — This repository serves as a comprehensive library of architectural blueprints and code examples for integrating large language models into software applications. It functions as a developer learning resource, providing structured tutorials and implementation patterns that demonstrate how to build intelligent features using advanced prompting and data processing techniques.

The collection distinguishes itself by focusing on complex reasoning and data-grounding workflows. It provides practical guidance on implementing retrieval-augmented generation pipelines, which connect language models to pr
- [danistefanovic/build-your-own-x](https://awesome-repositories.com/repository/danistefanovic-build-your-own-x.md) (516,495 ⭐) — Master programming by recreating your favorite technologies from scratch.
- [datawhalechina/all-in-rag](https://awesome-repositories.com/repository/datawhalechina-all-in-rag.md) (3,989 ⭐) — This project is a retrieval augmented generation framework designed to build pipelines that connect unstructured data and knowledge graphs with large language models. It functions as a vector database orchestrator for indexing text and multimodal content, as well as a system for translating natural language queries into structured database commands.

The framework integrates a hybrid retrieval engine that combines dense vector search with sparse keyword matching to increase the precision of retrieved contexts. It further enhances reasoning and relationship mapping through a graph-augmented ret
- [thoughtworks/build-your-own-radar](https://awesome-repositories.com/repository/thoughtworks-build-your-own-radar.md) (2,549 ⭐) — This project is a technology radar visualization tool and dockerized static site generator. It transforms JSON or CSV datasets into an interactive technology map used to track the adoption status and maturity of tools and techniques across an organization.

The tool enables enterprise architecture mapping by organizing portfolios of technologies into categories and maturity levels. It supports custom technical taxonomies, allowing the definition of specialized rings and quadrants to match specific organizational evaluation criteria.

The system covers automated radar generation and technology
- [firebase/genkit](https://awesome-repositories.com/repository/firebase-genkit.md) (6,121 ⭐) — Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management.

The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer
- [documentationjs/documentation](https://awesome-repositories.com/repository/documentationjs-documentation.md) (5,798 ⭐) — Documentation.js is a multi-purpose documentation tool that parses JSDoc annotations from JavaScript and TypeScript source files to generate formatted API documentation. It functions as both a documentation generator and a JSDoc linter, scanning source code for non-standard or incorrect annotations and returning human-readable warnings to enforce documentation quality.

The tool operates through a pipeline-based architecture that parses JSDoc comments into an abstract syntax tree, validates annotations against style and correctness rules, and outputs documentation through interchangeable plugi
- [pathwaycom/pathway](https://awesome-repositories.com/repository/pathwaycom-pathway.md) (62,959 ⭐) — Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources.

The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in
- [fredkschott/snowpack](https://awesome-repositories.com/repository/fredkschott-snowpack.md) (19,329 ⭐) — Snowpack is an ESM-powered frontend build tool and development server that serves native ES modules directly to the browser. By eliminating the bundling process during development, it enables nearly instant server startup and unbundled frontend development.

The project features a framework-aware hot module reload system that preserves component state during updates, with specific Fast Refresh integration for React, Preact, Svelte, and Vue. It also acts as a modern web transpiler, automatically converting TypeScript, JSX, and CSS Modules into browser-compatible code without requiring manual co
- [facebookresearch/parlai](https://awesome-repositories.com/repository/facebookresearch-parlai.md) (10,625 ⭐) — ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using a unified interface for datasets and agents. It functions as a PyTorch-based training platform and a dialogue data collection system, providing a centralized model zoo for the distribution of versioned pretrained agents.

The project distinguishes itself through a knowledge-grounded retrieval system that combines dense and sparse indexing to ground responses in external information. It also provides a comprehensive infrastructure for gathering human-AI interaction data via inte
- [postgresml/postgresml](https://awesome-repositories.com/repository/postgresml-postgresml.md) (6,801 ⭐) — PostgresML is a machine learning database extension for PostgreSQL that integrates model training and inference directly into the database. It functions as an in-database AI platform and vector database, enabling the execution of large language models and natural language processing tasks on stored records without exporting data to external services.

The system distinguishes itself by utilizing GPU acceleration to minimize latency during model predictions and employing a hybrid storage engine that maintains relational data alongside high-dimensional vectors. It allows for the building and fin
- [augmented-finance/augmented-finance-protocol](https://awesome-repositories.com/repository/augmented-finance-augmented-finance-protocol.md) (38 ⭐) — This repository contains the smart contracts source code and markets configuration for Augmented Finance Protocol.
- [dair-ai/prompt-engineering-guide](https://awesome-repositories.com/repository/dair-ai-prompt-engineering-guide.md) (75,678 ⭐) — This project is a comprehensive educational resource and technical guide focused on the development, optimization, and application of large language models. It provides a structured curriculum for mastering prompt engineering, ranging from foundational principles of instruction design to advanced techniques for improving model reasoning, accuracy, and reliability.

The guide distinguishes itself by offering deep technical insights into agentic workflows and autonomous system design. It covers the implementation of multi-step reasoning chains, tool integration through function calling, and stat
- [lukemathwalker/build-your-own-jira-with-rust](https://awesome-repositories.com/repository/lukemathwalker-build-your-own-jira-with-rust.md) (0 ⭐) — You will be working through a series of test-driven exercises, or koans, to learn Rust while building your own JIRA clone!
- [langchain-ai/rag-from-scratch](https://awesome-repositories.com/repository/langchain-ai-rag-from-scratch.md) (7,393 ⭐) — This project is an educational implementation guide and framework for building Retrieval Augmented Generation systems. It provides a workflow for constructing a knowledge base pipeline that partitions documents, indexes them as vectors, and provides external context for language model prompts.

The system features a document chunking framework that uses recursive character splitting to fit text into model context windows. It includes an in-memory vector store and a similarity search system that retrieves relevant text segments by calculating the mathematical distance between dense embedding ve
- [kutaytire/retrieval-augmented-time-series-forecasting](https://awesome-repositories.com/repository/kutaytire-retrieval-augmented-time-series-forecasting.md) (58 ⭐) — To perform inference for RAF and the baseline, install the necessary packages by running:
- [dragonflydb/dragonfly](https://awesome-repositories.com/repository/dragonflydb-dragonfly.md) (30,688 ⭐) — Dragonfly is a high-performance, multi-model in-memory data store designed to serve as a drop-in replacement for existing database infrastructures. By utilizing a multi-threaded, shared-nothing architecture and a fiber-based concurrency model, it maximizes CPU utilization and minimizes latency for read and write operations. The system supports a wide range of data structures, including strings, hashes, lists, sets, sorted sets, and JSON documents, while maintaining full compatibility with standard industry wire protocols and client libraries.

What distinguishes Dragonfly is its focus on effic
- [run-llama/rags](https://awesome-repositories.com/repository/run-llama-rags.md) (6,540 ⭐) — Rags is an orchestration tool for building retrieval-augmented generation pipelines and managing conversational data interfaces. It serves as a system for creating these pipelines from local files and web pages using natural language instructions to query, retrieve, and summarize information from connected datasets.

The project features a multimodal retrieval system that identifies and extracts information across different data types and modalities. It includes a vector search orchestrator to manage chunking strategies and search parameters, alongside a pipeline builder that translates conver
- [rom1504/clip-retrieval](https://awesome-repositories.com/repository/rom1504-clip-retrieval.md) (2,774 ⭐) — Easily compute clip embeddings and build a clip retrieval system with them
- [53ai/53aihub](https://awesome-repositories.com/repository/53ai-53aihub.md) (9,025 ⭐) — 53AIHub is a centralized orchestration platform for deploying and managing AI agents and prompts across multiple large language model providers. It functions as a multi-model AI gateway and an operation portal for AI services, providing a unified interface to coordinate agents and prompts from various external platforms.

The project distinguishes itself as a white-label AI portal designed for self-hosted infrastructure, allowing for full control over operational data on private servers or containers. It includes a comprehensive AI SaaS administration layer with a multi-tenant subscription eng
- [microsoft/promptflow](https://awesome-repositories.com/repository/microsoft-promptflow.md) (11,165 ⭐) — Promptflow is a development framework and orchestrator for building applications powered by large language models. It functions as a suite of tools for designing, orchestrating, and deploying AI workflows by linking prompts, custom Python code, and language models into executable sequences.

The project is distinguished by a visual AI workflow designer that allows for the creation of directed acyclic graphs of logic nodes. It provides a dedicated prompt engineering environment for versioning and comparing templates, alongside stateful execution tracing to record function calls and variable val
- [angular/angular](https://awesome-repositories.com/repository/angular-angular.md) (100,360 ⭐) — Angular is a platform for building web applications using a component-based architecture. It provides a comprehensive suite of tools for managing encapsulated UI units, including hierarchical dependency injection, a declarative template system, and fine-grained reactivity through signals. The framework supports complex application requirements such as client-side routing, form management, and internationalization.

The project includes a command-line interface for scaffolding and build automation, alongside a testing ecosystem for unit and integration verification. It offers multiple rendering
- [tokenrove/build-your-own-shell](https://awesome-repositories.com/repository/tokenrove-build-your-own-shell.md) (496 ⭐) — Guidance for mollusks (WIP)
- [tektoncd/pipeline](https://awesome-repositories.com/repository/tektoncd-pipeline.md) (8,996 ⭐) — Pipeline is a Kubernetes native CI/CD framework and cloud native pipeline orchestrator. It functions as a custom resource controller that translates declarative pipeline definitions into coordinated pod executions and managed workloads.

The system acts as a containerized task runner, allowing for the execution of standalone build steps and reusable tasks that process specific inputs to produce defined outputs. It enables the orchestration of complex workflows by running a sequence of independent containers as modular components within a cloud environment.

The platform covers automated softwa
- [google/clusterfuzz](https://awesome-repositories.com/repository/google-clusterfuzz.md) (5,574 ⭐) — ClusterFuzz is an automated platform that runs coverage-guided fuzzers at scale to find security and stability bugs in software. It orchestrates libFuzzer and AFL++ across distributed clusters of worker bots, collecting coverage feedback to guide input mutation and discover crashes. The platform provides a web-based dashboard for configuring fuzzing jobs, monitoring progress, and inspecting crash reports, with role-based access control to restrict sensitive features.

The system automates the full fuzzing lifecycle, from build pipeline integration and corpus management to crash triage and bug
- [embedchain/embedchain](https://awesome-repositories.com/repository/embedchain-embedchain.md) (58,769 ⭐) — Embedchain is an LLM memory management framework and RAG orchestration engine designed to provide AI agents with a persistent storage layer. It functions as a long-term memory pipeline that extracts facts from unstructured interactions and stores them as permanent knowledge base entries to retain user preferences and interaction history across sessions.

The system employs a hybrid vector database interface that combines semantic embeddings with traditional keyword search. It utilizes an entity-linking knowledge graph to connect related information points and applies temporal ranking to distin
- [tobi/qmd](https://awesome-repositories.com/repository/tobi-qmd.md) (9,498 ⭐) — qmd is a local semantic search engine and RAG knowledge base indexer that functions as a Model Context Protocol server. It converts local documents, markdown files, and codebases into a searchable database to provide retrieval augmented generation capabilities for AI agents.

The system exposes its search and retrieval tools via stdio or HTTP. It utilizes local model files for embeddings and reranking, supporting query expansion across multiple languages.

The project employs abstract syntax tree based chunking to split source code at function and class boundaries. It implements hybrid vector-
- [thephpleague/pipeline](https://awesome-repositories.com/repository/thephpleague-pipeline.md) (1,000 ⭐) — League\Pipeline
- [datalab-to/marker](https://awesome-repositories.com/repository/datalab-to-marker.md) (36,137 ⭐) — Marker is a comprehensive document processing platform designed to automate the conversion, extraction, and structuring of data from complex files. It functions as an orchestration engine that chains modular processing steps into versioned, reusable pipelines, allowing organizations to standardize document handling and automate repetitive business tasks at scale.

The platform distinguishes itself through its support for secure, private infrastructure deployment, enabling users to run containerized services within their own environments to maintain strict data privacy. It features specialized
- [gaia-pipeline/gaia](https://awesome-repositories.com/repository/gaia-pipeline-gaia.md) (5,216 ⭐) — Build powerful pipelines in any programming language.
- [meta-llama/llama-recipes](https://awesome-repositories.com/repository/meta-llama-llama-recipes.md) (18,379 ⭐) — This project is a collection of reference implementations and recipes for deploying, fine-tuning, and running inference with Llama large language models. It serves as a toolkit and implementation guide for adapting pre-trained models to specific tasks and domain-specific datasets.

The repository provides frameworks for developing retrieval augmented generation pipelines to ground model responses in external data. It includes guides for executing quantized inference to reduce memory usage and increase processing speed.

The toolkit covers a broad range of capabilities including parameter-effic
- [datalab-to/surya](https://awesome-repositories.com/repository/datalab-to-surya.md) (20,889 ⭐) — Surya is a document processing platform designed to transform unstructured files into structured, machine-readable data. It provides a comprehensive suite of tools for text recognition, layout analysis, and reading order detection, enabling the conversion of PDFs and images into formats such as JSON, HTML, or markdown. The platform is built to handle complex document workflows, offering capabilities for data extraction, document segmentation, and automated form completion.

The platform distinguishes itself through a robust pipeline-based architecture that allows users to chain analysis tasks
- [ardanlabs/gotraining](https://awesome-repositories.com/repository/ardanlabs-gotraining.md) (12,212 ⭐) — This repository provides curated learning paths, structured courseware, and technical materials for mastering Go programming, container orchestration, and software architecture. It serves as a comprehensive educational resource for systems programming, focusing on language mechanics, memory safety, and high-performance backend design.

The project distinguishes itself through a multi-modal instructional design that combines instructor-led workshops, project-based curricula, and competency-based certifications. It offers specialized guidance on building production-grade AI infrastructure, inclu
- [camel-ai/camel](https://awesome-repositories.com/repository/camel-ai-camel.md) (17,253 ⭐) — This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer.

The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-eva
- [lotabout/let-s-build-a-compiler](https://awesome-repositories.com/repository/lotabout-let-s-build-a-compiler.md) (580 ⭐) — A C & x86 version of the "Let's Build a Compiler" by Jack Crenshaw
- [the-pocket/pocketflow](https://awesome-repositories.com/repository/the-pocket-pocketflow.md) (10,046 ⭐) — PocketFlow is a graph-based framework for designing and executing large language model operations and reasoning patterns. It serves as an orchestrator for building goal-oriented autonomous agents, multi-agent systems, and retrieval-augmented generation pipelines.

The system is distinguished by its ability to coordinate autonomous AI agents that use shared memory and tools to solve complex goals, supported by a structured output engine that enforces schema-consistent responses. It utilizes graph-based workflow orchestration to manage sequences of model operations and supports supervisor-based
- [dubinc/dub](https://awesome-repositories.com/repository/dubinc-dub.md) (23,722 ⭐) — This project is a comprehensive link management and marketing attribution platform designed for creating, tracking, and analyzing shortened URLs. It functions as a centralized hub for marketing analytics, providing tools to monitor link performance, visualize conversion funnels, and manage affiliate programs through a unified dashboard.

The platform distinguishes itself by integrating advanced attribution modeling and partner management directly into the link infrastructure. It supports complex marketing workflows, including automated commission calculations, fraud detection, and payout distr
- [hyfather/pipeline](https://awesome-repositories.com/repository/hyfather-pipeline.md) (61 ⭐) — Pipelines using goroutines
- [0xplaygrounds/rig](https://awesome-repositories.com/repository/0xplaygrounds-rig.md) (7,450 ⭐) — Rig is a framework for building large language model applications, featuring a multi-provider client and a workflow builder for retrieval-augmented generation systems. It serves as an orchestrator for creating autonomous agents that can maintain conversation state and execute complex tasks through custom prompting and plugins.

The project provides standardized interfaces for both completion and embedding model providers, allowing for unified request and response patterns across different engines. It also includes a vector database integration layer that defines a common interface for indexing