Marker-Inc-Korea/AutoRAG
AutoRAG
AutoRAG ist eine Automatisierungsschicht und ein Optimierungstool für Retrieval-Augmented Generation (RAG). Es bietet ein Framework zur Messung der Pipeline-Performance durch ein Bewertungssystem sowie eine automatisierte Suchstrategie, die die effektivsten Kombinationen aus Retrieval- und Generierungsmodulen identifiziert.
Das System zeichnet sich durch AutoML-artige Optimierung aus, wobei Hyperparameter-Rastersuchen und automatisierte Testläufe verwendet werden, um die leistungsfähigste Architekturkonfiguration für einen spezifischen Datensatz zu finden. Es enthält einen spezialisierten Datensatz-Generator, der synthetische Frage-Antwort-Paare und Ground-Truth-Korpora aus Rohdokumenten erstellt, um Benchmarks für Tests zu etablieren.
Das Projekt deckt den gesamten RAG-Lebenszyklus ab, einschließlich multimodaler Dokumentenanalyse mit OCR und Tabellenerkennung, semantischem Chunking von Dokumenten und metrikbasierter Knotenbewertung. Es enthält zudem Hilfsmittel zur Visualisierung von Testergebnissen und zum Export optimierter Konfigurationen für die Bereitstellung als Skripte oder API-Server.
Die Pipeline wird über YAML-Konfigurationsdateien mit Unterstützung für die Injektion von Umgebungsvariablen verwaltet.
Features
- Retrieval Augmented Generation Pipelines - Provides a modular framework for constructing and optimizing end-to-end retrieval-augmented generation pipelines.
- RAG Pipeline Optimizers - Provides tools for automating the search and selection of the most effective retrieval and generation modules for RAG pipelines.
- RAG Evaluation Dataset Generation - Automatically generates question-answer pairs and ground truth from knowledge sources to benchmark RAG performance.
- RAG Evaluation Dataset Generators - Generates synthetic question-answer pairs from raw documents to establish ground truth benchmarks for evaluation.
- LLM Test Pair Generators - Generates synthetic question-answer pairs from document corpora using language models to establish evaluation benchmarks.
- Document Chunking Strategies - Divides parsed documents into segments using semantic or fixed-length strategies to optimize retrieval performance.
- RAG Pipelines - Allows connecting retrieval modules, prompt builders, and generators in a sequential chain.
- Document Chunking & Embedding - Splits extracted text into optimized segments and manages document preparation for retrieval-augmented generation.
- Automated ML (AutoML) - Implements automated selection and hyperparameter tuning of RAG modules to identify the most effective architectural configuration.
- RAG Architecture Optimization - Implements an automation layer that applies machine learning search strategies to optimize RAG architectures.
- RAG Evaluation Frameworks - Provides a framework for measuring RAG pipeline performance using specialized retrieval and generation metrics.
- Synthetic Dataset Generation - Generates synthetic question-answer pairs and ground truth corpora from raw documents for pipeline benchmarking.
- Document Parsing and Extraction - Converts various file formats like PDF and HTML into structured text suitable for LLM ingestion.
- Document Parsing Pipelines - Implements automated routines to parse diverse file formats like PDF and XML into structured text chunks.
- Document Parsing Engines - Integrates multiple OCR and text extraction engines to convert multimodal documents into structured formats.
- Structured Text Converters - Converts raw documents into metadata-rich text, Markdown, or JSON formats optimized for LLM analysis.
- Node Performance Evaluators - Provides metric-based evaluation of individual pipeline components to isolate performance bottlenecks.
- RAG Performance Benchmarks - Measures RAG workflow accuracy using specialized retrieval and generation metrics to identify optimal hyperparameters.
- RAG Configuration Experiments - Runs automated experiments across various module combinations to find the best configuration for a specific dataset.
- Grounded Answer Generation - Creates reference answers for questions based on provided context using a language model to establish baselines.
- Evaluation Visualizers - Displays trial outcomes and performance metrics through a dashboard to analyze the optimization process.
- Grid Search Executors - Performs exhaustive grid searches over predefined module versions and settings to optimize the retrieval-augmented generation pipeline.
- Multimodal Document Parsers - Uses vision-capable models to parse complex documents by analyzing visual elements alongside text.
- Custom Sentence Splitting - Integrates external tokenizers and custom splitting logic to support specialized text segmentation and multiple languages.
- RAG Performance Metrics - Sets specific performance thresholds and metrics to determine the effectiveness of individual pipeline nodes.
- OCR Document Parsers - Utilizes optical character recognition services to extract text from raw document images page by page.
- Hybrid Table Detectors - Combines text and table parsing modules to detect tabular data on a per-page basis.
- RAG Production Deployments - Allows exporting verified pipeline configurations into API servers or scripts for real-world production use.
- Configuration-Driven Pipelines - Uses YAML configuration files to define pipeline structures and parameters for reproducible evaluation and deployment.
- Pipeline Performance Evaluators - Runs automated trials against datasets to measure retrieval and generation metrics for configuration identification.
- Application Development - AutoML tool for optimizing RAG answer quality.
- Automated Machine Learning - AutoML tool specifically for optimizing RAG pipeline performance.
- Knowledge Retrieval - AutoML tool for finding optimal RAG pipelines.
- Retrieval Augmented Generation - AutoML tool for finding optimal retrieval pipeline configurations.
