nmslib/hnswlib
Hnswlib
hnswlib est une bibliothèque C++ header-only et un moteur d'indexation vectorielle conçu pour la recherche approximative de plus proches voisins en haute dimension. Il organise de grandes collections d'embeddings dans une structure de graphe interrogeable pour permettre des requêtes de proximité rapides et des calculs de distance.
Le système utilise des graphes Hierarchical Navigable Small World pour obtenir une recherche de similarité vectorielle rapide. Il se distingue en permettant la définition de métriques de distance et de fonctions de similarité personnalisées pour adapter les calculs à des exigences de données spécifiques.
Le moteur couvre tout le cycle de vie de l'indexation, incluant la construction incrémentale de l'index et la gestion des points de données via des ajouts et des suppressions d'éléments. Les capacités de requête incluent à la fois la recherche approximative et exacte des plus proches voisins, complétées par un filtrage de recherche booléen pour exclure des candidats basés sur des étiquettes d'éléments.
La bibliothèque prend en charge la persistance de l'index via la sérialisation de fichiers binaires et fournit des configurations pour l'exécution parallèle afin de distribuer les tâches de requête et d'indexation sur plusieurs cœurs CPU.
Features
- Hierarchical Proximity Graphs - Utilizes Hierarchical Navigable Small World graphs to achieve logarithmic search complexity through coarse-to-fine navigation.
- Vector Indexing - Implements a high-performance indexing engine specifically designed for high-dimensional vector data and similarity search.
- Vector Similarity Search - Implements algorithms and data structures for performing similarity search on high-dimensional vector embeddings.
- Approximate Nearest Neighbor Search - Utilizes Hierarchical Navigable Small World graphs to trade absolute precision for significantly faster proximity lookups.
- Graph - Constructs hierarchical proximity graphs by configuring link density to balance memory usage and retrieval recall.
- Vector Indexing Engines - Acts as a high-performance engine for organizing vectors to enable rapid proximity queries and distance calculations.
- HNSW Indexes - Provides a high-performance implementation of Hierarchical Navigable Small World (HNSW) indexes for approximate nearest neighbor search.
- Incremental Vector Sync - Supports incremental index management by adding or removing elements without requiring a full rebuild.
- Nearest Neighbor Searches - Implements algorithms for finding the closest vectors in high-dimensional space based on distance calculations.
- Vector Distance Metrics - Provides the mathematical framework for implementing specialized similarity functions between high-dimensional vectors.
- Customizable Distance Metrics - Allows the definition of custom similarity functions to adapt distance calculations to specific data requirements.
- Greedy Graph Search - Implements greedy search traversal to efficiently navigate the graph layers toward the closest vector.
- Header-only Libraries - Distributed as a header-only C++ library to enable compiler inlining and simplify integration.
- Search Index Deletions - Allows for the removal of specific data points from the search index via unique identifiers.
- Incremental Indexing Engines - Supports adding and updating vectors within the index without necessitating a full rebuild of the graph structure.
- Index Persistence Utilities - Enables saving and loading the current index state to binary files for persistence across sessions.
- Search Index Management - Includes tools for managing the index lifecycle, including marking elements as deleted to reclaim memory.
- Dynamic Index Updating - Provides mechanisms to update index content and replace elements to ensure the dataset remains current.
- Vector Databases - Provides the core indexing and retrieval layer necessary for building high-performance vector databases.
- Multi-threaded Execution - Distributes independent search and indexing requests across multiple CPU cores to increase throughput.
- Template-Based Metrics - Uses C++ template polymorphism to inject custom similarity functions without the overhead of virtual function calls.
- AI & Machine Learning - Fast approximate nearest neighbors library
- Machine Learning - Fast approximate nearest neighbor search library.
