# RAG framework Python

> Search results for `rag pipeline python` on awesome-repositories.com. 110 total matches; showing the first 50.

Explore on the web: https://awesome-repositories.com/q/rag-pipeline-python

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [this search on awesome-repositories.com](https://awesome-repositories.com/q/rag-pipeline-python).**

## Results

- [chonkie-inc/chonkie](https://awesome-repositories.com/repository/chonkie-inc-chonkie.md) (4,170 ⭐) — Chonkie is a text chunking library designed for retrieval-augmented generation pipelines. It functions as a semantic text splitter and RAG ingestion pipeline, transforming raw text into embedded segments for storage in vector databases.

The project distinguishes itself through specialized splitting strategies, including an AST-based code splitter for preserving logical boundaries in source code and a semantic text splitter that uses embedding models to determine boundaries based on meaning. It also provides a vector database ingestor to automate the generation of embeddings and their export t
- [maiot-io/zenml](https://awesome-repositories.com/repository/maiot-io-zenml.md) (5,452 ⭐) — ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments.

The project distinguishes itself
- [jamwithai/production-agentic-rag-course](https://awesome-repositories.com/repository/jamwithai-production-agentic-rag-course.md) (6,972 ⭐) — This project is an educational course and technical blueprint for building production-ready retrieval-augmented generation systems. It provides a curriculum and implementation strategies for designing agentic workflows, containerized AI infrastructure, and retrieval pipelines using large language models.

The materials focus on agentic design patterns, utilizing state-based decision nodes to rewrite queries and grade retrieved documents. It differentiates its approach by providing a deployment framework for managing databases, search engines, and API services through container orchestration.
- [kubeflow/pipelines](https://awesome-repositories.com/repository/kubeflow-pipelines.md) (4,154 ⭐) — This project is a containerized machine learning workflow engine and orchestrator designed to automate the end-to-end lifecycle of machine learning models on Kubernetes clusters. It functions as an MLOps pipeline compiler that transforms a domain-specific language into structured specifications for portable and scalable deployment.

The platform provides a multi-tenant environment with isolated namespaces and identity provider authentication. It distinguishes itself through a combination of container-based task isolation, strongly typed artifact management for data passing, and content-address
- [run-llama/rags](https://awesome-repositories.com/repository/run-llama-rags.md) (6,540 ⭐) — Rags is an orchestration tool for building retrieval-augmented generation pipelines and managing conversational data interfaces. It serves as a system for creating these pipelines from local files and web pages using natural language instructions to query, retrieve, and summarize information from connected datasets.

The project features a multimodal retrieval system that identifies and extracts information across different data types and modalities. It includes a vector search orchestrator to manage chunking strategies and search parameters, alongside a pipeline builder that translates conver
- [dair-ai/prompt-engineering-guide](https://awesome-repositories.com/repository/dair-ai-prompt-engineering-guide.md) (75,678 ⭐) — This project is a comprehensive educational resource and technical guide focused on the development, optimization, and application of large language models. It provides a structured curriculum for mastering prompt engineering, ranging from foundational principles of instruction design to advanced techniques for improving model reasoning, accuracy, and reliability.

The guide distinguishes itself by offering deep technical insights into agentic workflows and autonomous system design. It covers the implementation of multi-step reasoning chains, tool integration through function calling, and stat
- [langchain-ai/rag-from-scratch](https://awesome-repositories.com/repository/langchain-ai-rag-from-scratch.md) (7,393 ⭐) — This project is an educational implementation guide and framework for building Retrieval Augmented Generation systems. It provides a workflow for constructing a knowledge base pipeline that partitions documents, indexes them as vectors, and provides external context for language model prompts.

The system features a document chunking framework that uses recursive character splitting to fit text into model context windows. It includes an in-memory vector store and a similarity search system that retrieves relevant text segments by calculating the mathematical distance between dense embedding ve
- [kiln-ai/kiln](https://awesome-repositories.com/repository/kiln-ai-kiln.md) (4,910 ⭐) — Kiln is an LLM development workbench and evaluation framework designed for designing, testing, and optimizing prompts and AI agents. It functions as a multi-agent orchestrator and a RAG optimization tool, providing a visual interface for the iterative development of AI systems.

The project distinguishes itself through a comprehensive fine-tuning pipeline that supports zero-code model training and reasoning distillation. It enables the creation of hierarchical multi-agent systems where specialized actors coordinate via tool calling, and it implements a Model Context Protocol server to expose t
- [hkuds/rag-anything](https://awesome-repositories.com/repository/hkuds-rag-anything.md) (21,372 ⭐) — RAG-Anything is a retrieval-augmented generation framework designed to index diverse document formats and perform semantic search using local machine learning models. It functions as a local multimodal data processor, extracting and organizing information from various file types into a unified knowledge base to facilitate private document analysis.

The system distinguishes itself through its high-throughput ingestion engine, which processes large batches of documents into searchable vector embeddings. By executing machine learning models directly on local hardware, the framework ensures that
- [jenkinsci/pipeline-examples](https://awesome-repositories.com/repository/jenkinsci-pipeline-examples.md) (4,296 ⭐) — This project is a library of version-controlled workflow definitions and a collection of Groovy scripts and configuration snippets for implementing continuous integration and delivery automation in Jenkins. It serves as a reference for building automated pipelines using both declarative syntax and scripted logic.

The repository provides template collections and implementation patterns for creating software build and deployment workflows. It includes reusable functions and logic patterns designed to standardize pipeline behavior and eliminate code duplication across multiple projects through t
- [thephpleague/pipeline](https://awesome-repositories.com/repository/thephpleague-pipeline.md) (1,000 ⭐) — League\Pipeline
- [hyfather/pipeline](https://awesome-repositories.com/repository/hyfather-pipeline.md) (61 ⭐) — Pipelines using goroutines
- [datajuicer/data-juicer](https://awesome-repositories.com/repository/datajuicer-data-juicer.md) (6,574 ⭐) — Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines.

The project distinguishes itself through a YAML-based data recipe sys
- [letta-ai/letta](https://awesome-repositories.com/repository/letta-ai-letta.md) (21,168 ⭐) — Letta is a framework for building, deploying, and managing autonomous AI agents that maintain persistent state across long-term interactions. It provides a comprehensive suite of primitives for defining agents with configurable personas, modular memory blocks, and tool-use capabilities, enabling them to retain user preferences and conversation history over extended sessions.

The platform distinguishes itself through its advanced memory management and orchestration capabilities. It allows agents to autonomously update their own memory, perform retrieval-augmented generation, and coordinate com
- [infiniflow/ragflow](https://awesome-repositories.com/repository/infiniflow-ragflow.md) (82,922 ⭐) — This project is a comprehensive retrieval-augmented generation platform designed for building, managing, and deploying knowledge-based AI applications. It provides a unified environment for organizing datasets, configuring conversational chat assistants, and developing autonomous agents that execute multi-step reasoning workflows. By integrating document intelligence with advanced retrieval pipelines, the platform enables the creation of grounded, verifiable responses supported by traceable citations.

The platform distinguishes itself through deep document understanding and sophisticated know
- [intellabs/rag-fit](https://awesome-repositories.com/repository/intellabs-rag-fit.md) (770 ⭐) — Framework for enhancing LLMs for RAG tasks using fine-tuning.
- [ardanlabs/service](https://awesome-repositories.com/repository/ardanlabs-service.md) (4,030 ⭐) — This project provides a set of structural templates and frameworks for bootstrapping production servers, high-performance backends, Kubernetes microservices, and AI pipelines using the Go programming language. It serves as a foundational architecture for building high-throughput infrastructure and scalable production servers with integrated routing and middleware.

The framework includes a specialized infrastructure for developing retrieval-augmented generation systems, emphasizing local model inference and secure data sovereignty. It further provides a dedicated microservice template for cont
- [cmavro/gnn-rag](https://awesome-repositories.com/repository/cmavro-gnn-rag.md) (435 ⭐) — This is the code for GNN-RAG: Graph Neural Retrieval for Large Language Modeling Reasoning.
- [containers/ramalama](https://awesome-repositories.com/repository/containers-ramalama.md) (2,605 ⭐) — Ramalama is a containerized runtime and management tool for large language models. It functions as an OCI AI model manager and registry client, allowing users to package, distribute, and execute AI models as standardized container images.

The project differentiates itself by using OCI-compliant distribution for models and retrieval augmented generation assets, enabling the packaging of vector databases into immutable container images. It features hardware-aware image selection that automatically detects GPU or CPU capabilities to pull the most optimized image for the host environment.

The sy
- [flowiseai/flowise](https://awesome-repositories.com/repository/flowiseai-flowise.md) (53,641 ⭐) — Flowise is a low-code platform designed for building and deploying complex language model workflows through a visual, node-based interface. It functions as an orchestrator for autonomous multi-agent systems, allowing users to construct conversational pipelines by connecting language models, memory stores, and external tools on a drag-and-drop canvas.

The platform distinguishes itself through its support for sophisticated agentic patterns, including supervisor-worker delegation and iterative reasoning strategies. Users can design directed acyclic graphs to manage conditional branching, state p
- [deepset-ai/haystack](https://awesome-repositories.com/repository/deepset-ai-haystack.md) (24,253 ⭐) — Haystack is an orchestration framework designed for building complex search and generative AI pipelines. It functions as an agentic workflow engine, enabling the construction of automated sequences that allow AI agents to perform multi-step reasoning and data analysis.

The framework utilizes a modular, component-based architecture that connects processing steps into directed acyclic graphs. By employing a provider-agnostic integration layer, it decouples core logic from specific external AI services and vector databases, allowing for the flexible exchange of underlying technologies. This desi
- [mrrezaeiuoft/amg-rag](https://awesome-repositories.com/repository/mrrezaeiuoft-amg-rag.md) (34 ⭐) — AMG-RAG (Agentic Medical Graph-RAG) is a comprehensive framework that automates the construction and continuous updating of Medical Knowledge Graphs (MKGs), integrates reasoning, and retrieves current external evidence for medical Question Answering (QA). Our approach addresses the challenge of…
- [yusufkaraaslan/skill_seekers](https://awesome-repositories.com/repository/yusufkaraaslan-skill-seekers.md) (9,641 ⭐) — Skill Seekers is a toolset for generating large language model knowledge bases, featuring a multi-source content scraper and a dedicated RAG data pipeline. It extracts technical data from documentation, code, and video to create structured assets and configuration files for AI-powered IDE extensions.

The project distinguishes itself through the ability to transform raw data into polished tutorials and specialized skills for AI plugin marketplaces. It utilizes abstract syntax tree parsing and optical character recognition to analyze GitHub repositories, PDFs, and video frames, converting these
- [goodnight77/just-rag](https://awesome-repositories.com/repository/goodnight77-just-rag.md) (135 ⭐) — A collection of some RAG tutorials where i share what i learned in my own way.
- [lazyagi/lazyllm](https://awesome-repositories.com/repository/lazyagi-lazyllm.md) (3,842 ⭐) — LazyLLM is a multi-agent framework and orchestration engine designed for building complex AI applications. It provides a system for chaining large language models into sequential or parallel pipelines, utilizing a tool registry to convert standard functions into discoverable tools that models can invoke via reasoning.

The project features an application deployment kit that enables hosting model workflows as web services with integrated chat interfaces and API gateways. It includes an infrastructure abstraction layer that allows users to switch between bare-metal servers, clusters, and public
- [quivrhq/megaparse](https://awesome-repositories.com/repository/quivrhq-megaparse.md) (7,389 ⭐) — Megaparse is a document parsing tool and RAG data preprocessor designed to convert PDFs, Word documents, and presentations into clean text formats. It functions as a vision-based document extractor that recovers high-fidelity information from images and complex layouts to optimize data for large language model ingestion.

The system employs multimodal AI and vision models to perform schema-preserving parsing, which maintains structural hierarchies such as tables and headers. It utilizes lossless structural transformation to turn layout-heavy binary files into text sequences while preserving th
- [tektoncd/pipeline](https://awesome-repositories.com/repository/tektoncd-pipeline.md) (8,996 ⭐) — Pipeline is a Kubernetes native CI/CD framework and cloud native pipeline orchestrator. It functions as a custom resource controller that translates declarative pipeline definitions into coordinated pod executions and managed workloads.

The system acts as a containerized task runner, allowing for the execution of standalone build steps and reusable tasks that process specific inputs to produce defined outputs. It enables the orchestration of complex workflows by running a sequence of independent containers as modular components within a cloud environment.

The platform covers automated softwa
- [camel-ai/camel](https://awesome-repositories.com/repository/camel-ai-camel.md) (17,253 ⭐) — This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer.

The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-eva
- [googlecloudplatform/agent-starter-pack](https://awesome-repositories.com/repository/googlecloudplatform-agent-starter-pack.md) (5,752 ⭐)
- [raudaschl/rag-fusion](https://awesome-repositories.com/repository/raudaschl-rag-fusion.md) (940 ⭐) — RAG-Fusion: multi-query generation + Reciprocal Rank Fusion for better retrieval-augmented generation. Includes evaluation harness with NFCorpus/BEIR.
- [banzaicloud/pipeline](https://awesome-repositories.com/repository/banzaicloud-pipeline.md) (1,504 ⭐) — Banzai Cloud Pipeline is a solution-oriented application platform which allows enterprises to develop, deploy and securely scale container-based applications in multi- and hybrid-cloud environments.
- [docling-project/docling](https://awesome-repositories.com/repository/docling-project-docling.md) (61,674 ⭐) — Docling is a modular framework designed for document parsing, layout analysis, and structured data extraction. It transforms unstructured files and web content into a unified, hierarchical data model that preserves the spatial and semantic relationships between text, tables, images, and layout elements. By normalizing diverse input formats into a consistent internal representation, the library enables uniform processing across various document types.

The project distinguishes itself through a schema-driven approach that maps document regions to strongly-typed objects, ensuring data accuracy t
- [stangirard/quiver](https://awesome-repositories.com/repository/stangirard-quiver.md) (39,167 ⭐) — Quiver is a framework for integrating retrieval augmented generation into applications. It provides a generative AI integration layer that connects large language models with vector stores to produce context-aware responses based on custom data.

The project features a knowledge base pipeline that parses diverse file types into searchable embeddings and a vector database orchestrator to manage data across different storage implementations. It utilizes a provider-agnostic model interface, allowing users to switch between various external AI providers or local models through a single unified sys
- [huggingface/smolagents](https://awesome-repositories.com/repository/huggingface-smolagents.md) (27,885 ⭐) — This framework provides a development toolkit for building autonomous agents that utilize language models to solve complex, non-deterministic tasks. Its core design centers on a code-executing architecture where agents generate and run Python code snippets to perform logic, data manipulation, and tool interactions. By moving beyond structured data formats, the system enables agents to manage program flow and object state through iterative reasoning cycles.

The project distinguishes itself through its focus on code-based agent implementation and secure execution environments. Developers can ch
- [apache/apisix](https://awesome-repositories.com/repository/apache-apisix.md) (16,767 ⭐) — This project is a high-performance, distributed API gateway designed to manage, secure, and observe traffic for microservices, serverless functions, and artificial intelligence model providers. It functions as a dynamic service proxy and cloud-native ingress controller, centralizing policy enforcement and traffic routing through a unified configuration interface that synchronizes state across multiple nodes in real time.

The platform distinguishes itself through a highly extensible architecture that utilizes a high-performance scripting engine to execute modular logic directly within the requ
- [alpacahq/pipeline-live](https://awesome-repositories.com/repository/alpacahq-pipeline-live.md) (207 ⭐) — Pipeline Extension for Live Trading
- [vikparuchuri/marker](https://awesome-repositories.com/repository/vikparuchuri-marker.md) (36,164 ⭐) — Marker is an LLM-powered document parser and OCR pipeline designed to convert PDFs and unstructured files into structured markdown, JSON, and HTML. It functions as a data preprocessor that transforms complex documents into machine-readable formats while preserving tables, equations, and layout structures.

The system utilizes large language models to refine OCR accuracy, clean mathematical notation, and merge fragmented tables across multiple pages. It employs model-based layout analysis to predict block types and bounding boxes, ensuring a more precise conversion of document elements.

Capabi
- [agentscope-ai/agentscope](https://awesome-repositories.com/repository/agentscope-ai-agentscope.md) (26,895 ⭐) — Agentscope is a comprehensive toolkit for developing and orchestrating autonomous multi-agent systems. It provides a unified framework for building agents that can reason, execute tools, and manage memory, enabling the creation of complex, collaborative workflows where multiple specialized agents interact to solve multi-step objectives.

The platform distinguishes itself through a robust orchestration engine that supports both sequential and concurrent agent pipelines. It utilizes a centralized event bus for real-time telemetry, allowing developers to track agent reasoning, tool usage, and sys
- [nkapila6/mcp-local-rag](https://awesome-repositories.com/repository/nkapila6-mcp-local-rag.md) (126 ⭐) — "primitive" RAG-like web search model context protocol (MCP) server that runs locally. ✨ no APIs ✨
- [ds4sd/docling](https://awesome-repositories.com/repository/ds4sd-docling.md) (62,172 ⭐) — Docling is a multimodal content converter and document parser designed to transform PDFs, Office files, and HTML into structured Markdown or JSON for generative AI applications. It functions as an OCR document processor and a PDF layout analyzer that extracts tables, charts, and hierarchical structures while preserving the original page layout.

The system operates as a local-first inference engine, allowing for the processing of sensitive data in air-gapped environments without external network connectivity. It can also be deployed as an API or a Model Context Protocol server to provide parsi
- [dataelement/bisheng](https://awesome-repositories.com/repository/dataelement-bisheng.md) (11,455 ⭐) — Bisheng is an enterprise AI framework and LLM DevOps platform designed to manage the full lifecycle of large language models. It provides a unified system for dataset curation, supervised fine-tuning, model versioning, and performance evaluation.

The platform features a visual workflow orchestrator for building retrieval-augmented generation pipelines and complex task sequences using flowcharts with conditional logic and human intervention points. It also includes an AI agent framework that uses a specialized guidance language to embed domain expertise and professional business logic into aut
- [toodef/neural-pipeline](https://awesome-repositories.com/repository/toodef-neural-pipeline.md) (312 ⭐) — Neural networks training pipeline based on PyTorch
- [datahub-project/datahub](https://awesome-repositories.com/repository/datahub-project-datahub.md) (12,141 ⭐) — DataHub is a metadata management platform designed to unify technical, operational, and business context across diverse data ecosystems. By utilizing a graph-based metadata model and an event-driven ingestion architecture, it creates a centralized source of truth that maps complex data relationships, lineage, and ownership. This foundational framework enables organizations to maintain a synchronized view of their data landscape, supporting both human-led discovery and automated data operations.

The platform distinguishes itself through its focus on grounding artificial intelligence and autono
- [shubhamsaboo/awesome-llm-apps](https://awesome-repositories.com/repository/shubhamsaboo-awesome-llm-apps.md) (114,725 ⭐) — This repository serves as a comprehensive collection of resources, templates, and starter code for building artificial intelligence applications. It provides a centralized hub for developers to access practical implementations of common workflows, including retrieval-augmented generation pipelines and autonomous agent loops, alongside educational materials designed to support rapid prototyping and experimentation.

The project distinguishes itself by offering a dual focus on technical implementation and critical analysis. It provides a library of lightweight, single-file agents and tutorials f
- [nxdir-s/pipelines](https://awesome-repositories.com/repository/nxdir-s-pipelines.md) (17 ⭐) — pipelines contains generic functions for concurrent processing
- [zenml-io/zenml](https://awesome-repositories.com/repository/zenml-io-zenml.md) (5,451 ⭐) — ZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure.

The platform distinguishes itself through a service-oriented
- [allaboutai-yt/easy-local-rag](https://awesome-repositories.com/repository/allaboutai-yt-easy-local-rag.md) (1,221 ⭐) — Easy Local RAG is a system for building and operating private, offline retrieval-augmented generation pipelines. It enables users to perform semantic search, document querying, and conversational analysis on local data sources without transmitting sensitive information to external cloud providers.

The project distinguishes itself by integrating specialized utilities for archiving personal email communications alongside standard document processing. By leveraging locally hosted language models and a local vector database, it maintains full control over data ingestion, indexing, and model infer
- [jazzband/django-pipeline](https://awesome-repositories.com/repository/jazzband-django-pipeline.md) (1,544 ⭐) — Pipeline is an asset packaging library for Django.
- [timescale/pgai](https://awesome-repositories.com/repository/timescale-pgai.md) (5,802 ⭐) — pgai is a PostgreSQL AI toolkit and framework designed to integrate large language models and vector embeddings directly into a database. It serves as a bridge for executing machine learning model requests and performing text-to-SQL translations within standard database queries.

The project provides an automated vector embedding pipeline that handles the loading, parsing, and chunking of text from tables and unstructured documents. This system utilizes a background worker to synchronize embeddings automatically as source data changes and includes specialized tools for building retrieval-augme
- [ionicfirebaseapp/getwidget](https://awesome-repositories.com/repository/ionicfirebaseapp-getwidget.md) (4,819 ⭐) — Getwidget is a comprehensive Flutter UI component library and design system providing over 1,000 pre-built widgets for creating cross-platform user interfaces. It serves as a production-ready widget gallery designed to accelerate the front-end development process through a wide array of pre-composed elements.

The project includes a specialized AI interaction UI kit featuring components for rendering streaming large language model responses, RAG chat interfaces, and real-time voice agents. It also provides operational utilities for estimating monthly LLM expenditures based on token usage and p