AnswerDotAI/RAGatouille
RAGatouille
RAGatouille is a retrieval framework and search engine designed for implementing and training late-interaction retrieval models. It serves as a modular retrieval component for generative AI pipelines, focusing on high-performance document ranking to improve search accuracy.
The project provides a toolkit for training and fine-tuning retrieval models using pairs and triplets, featuring automatic hard-negative mining for domain adaptation. It implements a late-interaction mechanism that balances retrieval speed with precision by utilizing compressed embeddings.
The system covers document indexing and retrieval operations, utilizing disk-based vector storage to handle datasets that exceed available system memory. It further supports the creation of retrieval augmented generation workflows by mapping token-level embeddings to preserve granular semantic information.
Features
- Late Interaction Retrieval - Implements a late-interaction mechanism that stores multiple vectors per document for granular semantic matching.
- Retrieval Model Fine-Tuning - Provides workflows for optimizing late-interaction retrieval models using query-passage pairs to improve search precision.
- RAG Context Retrieval - Retrieves relevant document segments from a knowledge base to provide grounded context for language model responses.
- Ranking Model Training - Trains retrieval models to order documents by relevance using a ranking objective based on positive and negative pairs.
- Hard Negative Mining - Automatically identifies challenging dissimilar examples during training to refine embedding boundaries and improve search precision.
- Semantic Word Embeddings - Maps every word in a document to a semantic vector to preserve granular information for late-interaction scoring.
- Late Interaction Search Engines - Builds search systems that utilize multi-vector matching to balance retrieval speed and ranking precision.
- NLP Model Trainers - Ships a toolkit for fine-tuning retrieval models using pairs, triplets, and automatic hard-negative mining.
- Vector Embedding Indexing - Tokenizes and embeds documents into compressed bags of embeddings stored on disk for fast retrieval.
- Vector Storage - Provides specialized storage engines that persist high-dimensional vectors to disk to handle datasets exceeding system memory.
- Vector Indexing Pipelines - Processes and embeds large text collections into compressed formats for efficient similarity search.
- Indexed Document Retrieval - Retrieves the most relevant documents from a vector index based on late-interaction scoring.
- Vector Index Compression - Implements quantized vector storage to reduce memory footprint and latency during retrieval.
