Photo Deduplication and Similarity Tools

LibrePhotos is a self-hosted photo management system designed for organizing, tagging, and searching photos and videos stored on personal servers. It functions as an AI-powered media organizer and a multi-user media server, providing isolated accounts and secure permissions so multiple people can manage private collections on a single system. The project differentiates itself through a semantic image search engine that allows for natural language queries based on deep metadata analysis. It incorporates computer vision to automatically detect faces and identify objects, while utilizing geospat

Czkawka is a cross-platform utility designed for storage optimization and filesystem maintenance. It functions as a comprehensive file analysis engine that identifies redundant data, including duplicate files, empty directories, broken symbolic links, and temporary files. By utilizing hash-based content verification, the tool ensures accurate identification of duplicates regardless of file names or metadata. The project distinguishes itself by offering both a native graphical user interface and a command-line interface, allowing for both interactive management and automated, headless system m

imagededup is a Python library used for finding exact and near-duplicate images. It provides utilities for generating image fingerprints, computing neural embeddings, and evaluating the precision of deduplication processes. The tool utilizes perceptual hashing to identify visually similar files regardless of size or format and employs deep learning models to encode images into vectors for high-accuracy similarity searches. It includes a system for measuring the precision and recall of these processes by comparing results against known ground truth datasets. The library covers broader capabil

dupeguru is a content-based file deduplicator and cross-platform disk cleanup tool. It functions as a local file management utility designed to identify and remove redundant files to recover disk space. The application identifies identical files across a file system by using content hashing and metadata comparison. This allows it to detect duplicates regardless of their filenames or directory locations. The software covers a range of data management capabilities, including directory scanning, content-based data deduplication, and the consolidation of redundant copies into single versions. It

PaddleClas is a toolkit for image classification and recognition built on PaddlePaddle. It provides a suite of tools for training deep learning models and a framework for implementing visual search and retrieval systems. The project includes a computer vision model optimization suite and tools for cross-platform deployment. It enables the export of trained models to servers, mobile devices, and edge hardware to achieve high-performance inference across different programming languages. The toolkit covers model compression and optimization through pruning, quantization, and knowledge distillat

This project is a C++ vector similarity engine and implementation of the Hierarchical Navigable Small World algorithm. It provides a header-only library for performing approximate nearest neighbor searches in high-dimensional spaces, alongside Python bindings that expose these indexing and search capabilities to data science environments. The engine enables real-time embedding retrieval and high-dimensional similarity search using a multi-layered graph structure to balance search speed and accuracy. It supports custom distance metrics to calculate similarity between vectors in various mathema

Caffe is a high-performance deep learning framework designed for training and deploying deep neural networks. It functions as a machine learning engine and a convolutional neural network library, providing a C++ backend to accelerate computations on both GPUs and CPUs. The system includes a specialized toolset for computer vision, enabling tasks such as object detection, semantic segmentation, and large-scale image retrieval. It supports the deployment of pre-trained models for image and scene recognition, as well as the ability to fine-tune neural network weights for specialized tasks. The

Embedding Atlas is a web-based interface for rendering high-dimensional vector embeddings and analyzing complex datasets through interactive visual clustering. It functions as a high-dimensional data analyzer used to discover trends and density patterns, acting as a vector similarity explorer to locate nearest neighbor data points within large-scale embedding datasets. The project provides a synchronized multimodal data dashboard that links tabular data with images, audio, and text. It utilizes hardware-accelerated rendering to display millions of embedding points and employs high-dimensional

This project is a collection of educational resources and implementation frameworks providing deep learning model recipes, code samples, and step-by-step guides for computer vision tasks. It organizes complex workflows into modular recipes and implementation guides to facilitate the building of image and video analysis models. The framework focuses on specialized vision capabilities, including an image similarity framework for fast retrieval and re-ranking, human pose estimation, and video action recognition. It also provides specific tools for crowd density estimation and document image clea

Marqo is an ecommerce product discovery platform, multimodal vector database, and AI search merchandising tool. It provides infrastructure for implementing semantic search and recommendations, allowing shoppers to find products using natural language and images. The platform distinguishes itself through a hybrid ranking pipeline that combines neural semantic scores with business-defined boosting and pinning rules. It features a conversational commerce engine that uses large language models to process user intent and provides a search performance analytics suite for measuring conversion uplift