datawhalechina/all-in-rag
All In Rag
This project is a retrieval augmented generation framework designed to build pipelines that connect unstructured data and knowledge graphs with large language models. It functions as a vector database orchestrator for indexing text and multimodal content, as well as a system for translating natural language queries into structured database commands.
The framework integrates a hybrid retrieval engine that combines dense vector search with sparse keyword matching to increase the precision of retrieved contexts. It further enhances reasoning and relationship mapping through a graph-augmented retrieval system.
The system includes a toolkit for measuring the quality of retrieval and generation processes using standardized metrics. It also provides mechanisms to enforce predefined schemas and patterns on model responses to ensure consistent output for downstream applications.
The project is implemented in Python.
Features
- RAG Pipelines - Provides a comprehensive framework for building retrieval-augmented generation pipelines that integrate external data sources.
- Graph Knowledge Indexing - Structures knowledge as interconnected nodes and relationships to improve reasoning and semantic retrieval.
- Graph Retrieval Augmented Generation - Combines knowledge graphs with retrieval-augmented generation to enhance reasoning and relationship mapping in responses.
- Hybrid Search Systems - Provides a search system that fuses multiple retrieval methods, including dense and sparse search.
- Knowledge Graph Retrieval Systems - Leverages graph topology and entity relationships to provide context-aware retrieval for generated responses.
- Knowledge Graphs - Integrates structured knowledge graphs to provide reasoning and context for generated model responses.
- RAG Frameworks - Acts as a comprehensive framework for building RAG pipelines that integrate unstructured data and knowledge graphs.
- Hybrid Sparse-Dense Embeddings - Integrates dense semantic vectors with sparse keyword-based representations to increase retrieval precision.
- Document and Unstructured Extraction - Implements automated parsing and splitting of unstructured documents and web content into machine-readable formats.
- Vector Store Orchestrators - Orchestrates the indexing of text and multimodal content into vector databases for high-performance retrieval.
- Hybrid Retrieval - Implements a hybrid retrieval engine combining dense vector similarity and sparse keyword matching.
- Vector Indexing - Implements high-dimensional vector indexing for semantic retrieval of unstructured and multimodal content.
- Model Output Formatting - Includes utilities for enforcing structured data schemas in language model responses.
- Output Formatting Constraints - Enforces specific output schemas and formats on language model responses to ensure consistency.
- RAG Evaluation Frameworks - Provides a toolkit for evaluating the performance of retrieval and generation processes in RAG workflows.
- Constrained Decoding - Uses constrained decoding and validation to force model outputs into predefined structured formats.
- Structured Output Enforcements - Enforces specific patterns and predefined schemas on language model responses to ensure output consistency.
- Text-to-SQL Translators - Translates natural language queries into structured SQL commands for precise data retrieval from relational databases.
- LLM Schema Outputs - Enforces predefined schemas and formats on language model responses for consistent downstream data processing.
- Multi-Stage Pipeline Processing - Orchestrates multi-stage pipelines that chain data loading, text chunking, and indexing into a sequential workflow.
- System Quality Evaluators - Provides a framework for applying custom metrics to quantify the performance of RAG workflows.
- Retrieval Metrics - Provides quantitative measurement of retrieval quality using standardized metrics against ground-truth datasets.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
