erikbern/ann-benchmarks
Ann Benchmarks
ann-benchmarks is a standardized benchmarking suite for evaluating approximate nearest neighbor (ANN) search algorithms. It provides a reproducible framework for comparing how different ANN implementations perform across multiple datasets and distance measures, using precomputed ground truth results stored in HDF5 files to ensure consistent accuracy measurements.
The project wraps each ANN library behind a common Python interface, allowing algorithms to be swapped without modifying the benchmarking harness. It orchestrates parameter sweeps over algorithm-specific hyperparameters to generate full speed-accuracy trade-off curves, and renders interactive scatter plots that visualize recall versus queries per second across different configurations. The suite includes a Dockerized execution environment that packages the entire benchmark pipeline, enabling identical results across different machines.
The framework supports algorithm verification through test suites that check correctness before benchmarking, and provides interactive result browsing with per-algorithm performance views, detailed metric inspection, and filtering by distance measure and dataset. It also accepts community algorithm contributions through pull requests, allowing new implementations to be added to the standardized comparison.
Features
- Benchmarking Suites - Provides a standardized benchmarking suite for evaluating approximate nearest neighbor search algorithms with reproducible speed-accuracy trade-off curves.
- Standardized ANN Benchmarks - Runs standardized speed-accuracy comparisons of ANN algorithms across multiple datasets and distance measures.
- Hyperparameter Sweep Orchestrators - Automates parallel execution of training runs with varying hyperparameters to generate speed-accuracy curves.
- Speed-Accuracy Tradeoff Comparisons - Runs multiple ANN implementations on standardized datasets and reviews their speed-accuracy trade-offs.
- Batch Query Execution - Runs all queries in a single batch against each index to measure throughput and latency.
- Ground Truth Storage - Stores precomputed nearest neighbor results in HDF5 files to enable reproducible accuracy measurements.
- Precomputed Ground Truth Datasets - Uses pre-split HDF5 datasets with ground truth top-100 neighbors to ensure reproducible results.
- Python Library Wrappers - Wraps each ANN library behind a common Python interface, enabling drop-in algorithm swapping without modifying the benchmarking harness.
- Dockerized Benchmark Runners - Packages the entire benchmark suite into Docker containers for reproducible execution across machines.
- ANN Algorithm Comparisons - Evaluates approximate nearest neighbor algorithms across multiple datasets and distance measures.
- Containerized ANN Benchmarks - Benchmarks ANN algorithms using Docker containers and precomputed HDF5 datasets with ground truth.
- Benchmark Result Plotters - Generates interactive scatter plots of recall versus queries per second for comparing ANN algorithm performance across configurations.
- Community Benchmark Contributions - Accepts community contributions to add new algorithms to the standardized benchmark suite.
- ANN Algorithm Contributions - Allows adding custom approximate nearest neighbor implementations to the standardized benchmarking framework.
- Algorithm Verification Testing - Runs test suites that check each nearest neighbor algorithm's function integrity before benchmarking.
- Interactive Benchmark Plots - Generates interactive scatter plots visualizing recall versus queries per second across algorithms and datasets.
- ANN Performance Visualizations - Provides interactive plots showing recall, index size, and build time trade-offs for nearest neighbor search.
- Interactive Metric Drill-Downs - Allows clicking on result charts to reveal interactive plots for detailed metric inspection.
- Per-Algorithm Result Views - Provides per-algorithm performance views across all datasets with interactive plots.
- Distance-Measure Filtered Comparisons - Filters benchmark results by distance measure and dataset to compare algorithm performance.
- Model Evaluation and Benchmarking - Benchmarking environment for approximate nearest neighbor search algorithms.
