Awesome GitHub RepositoriesVector Index Compression

ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using a unified interface for datasets and agents. It functions as a PyTorch-based training platform and a dialogue data collection system, providing a centralized model zoo for the distribution of versioned pretrained agents. The project distinguishes itself through a knowledge-grounded retrieval system that combines dense and sparse indexing to ground responses in external information. It also provides a comprehensive infrastructure for gathering human-AI interaction data via inte

zvec is an embedded vector database engine and indexing library designed for high-dimensional similarity search. It functions as a hybrid search engine and a retrieval-augmented generation knowledge base, allowing for the storage and retrieval of dense and sparse vectors. The system is distinguished by its hybrid retrieval pipeline, which fuses vector similarity, full-text keyword matching, and scalar metadata filtering into single query operations. It supports a plugin-based model integration system for registering custom embedding models and rerankers, as well as language bindings for nativ

RAGatouille ist ein Retrieval-Framework und eine Suchmaschine zur Implementierung und zum Training von Late-Interaction-Retrieval-Modellen. Es dient als modulare Retrieval-Komponente für generative KI-Pipelines und konzentriert sich auf hochperformantes Dokument-Ranking zur Verbesserung der Suchgenauigkeit. Das Projekt bietet ein Toolkit zum Training und Fine-Tuning von Retrieval-Modellen unter Verwendung von Paaren und Tripletts, inklusive automatischem Hard-Negative-Mining für die Domänenanpassung. Es implementiert einen Late-Interaction-Mechanismus, der Retrieval-Geschwindigkeit und Präzision durch die Verwendung komprimierter Embeddings ausbalanciert. Das System deckt Dokument-Indizierungs- und Retrieval-Operationen ab und nutzt disk-basierten Vektorspeicher, um Datensätze zu verarbeiten, die den verfügbaren Arbeitsspeicher übersteigen. Es unterstützt zudem die Erstellung von RAG-Workflows (Retrieval Augmented Generation), indem Token-Level-Embeddings gemappt werden, um granulare semantische Informationen zu bewahren.

USearch is a high-performance vector similarity search engine and approximate nearest neighbor index designed for dense embeddings. It functions as a low-level vector database core and high-dimensional vector indexer, providing the primitives necessary to store and retrieve vectors across massive datasets. The engine distinguishes itself through hardware-level SIMD acceleration for distance kernels and a proximity-graph indexing system that enables fast retrieval across billions of vectors. It supports multi-precision vector quantization to balance memory usage and accuracy, and utilizes memo