Awesome GitHub RepositoriesData Deduplication Tools

This project is a human resources management system built using Spring Boot and Vue. It serves as a platform for managing employee records, professional titles, and organizational hierarchies. The system features a role-based access control framework that maps users to specific roles and resources to secure API endpoints and user interface elements. It includes a real-time communication hub utilizing WebSockets for internal corporate chat and system notifications, as well as a dedicated manager for defining and modifying nested organizational department structures. Additional capabilities co

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

Memori is an AI agent memory middleware platform designed to provide persistent, context-aware recall for language models. It functions as a non-intrusive layer that intercepts outbound model requests to automatically capture interaction history and execution traces, ensuring that agents maintain continuity across sessions without requiring modifications to existing application logic. The platform distinguishes itself through a dual-model storage architecture that maintains information as both structured relational primitives for precise fact retrieval and rolling narrative summaries for situ

This repository is a comprehensive collection of instructional guides and practical examples for Python development, focusing on machine learning, data science, and web scraping. It provides implementations for neural networks, reinforcement learning algorithms, and deep learning architectures using PyTorch, alongside detailed manuals for scientific computing and data visualization. The project distinguishes itself by offering specialized tutorials on concurrent programming to optimize CPU performance and guides for setting up Linux development environments. It covers the implementation of ad

Toastr is a JavaScript toast notification library and client-side alert manager used to display non-blocking alert messages and system notifications in web browsers. It serves as a frontend notification UI for rendering timed messages that provide real-time visual feedback within web applications. The library includes a notification system that supports right-to-left layout mirroring for internationalized applications. It also features a mechanism to prevent duplicate notifications by matching new content against active alerts to avoid redundant stacking. The system manages notification life

Keep is an open-source AIOps alert management platform that aggregates, deduplicates, and orchestrates the lifecycle of alerts from multiple monitoring tools. It functions as a multi-provider integration hub to centralize the flow of data between observability, ticketing, and communication tools. The platform distinguishes itself through incident workflow automation and AI-powered enrichment. It uses a declarative workflow engine to execute multi-step operational sequences and integrates large language models to summarize event data and correlate technical logs for faster incident resolution.

BasicSR is a PyTorch-based image restoration toolbox and framework designed for training and deploying deep learning models to upscale, denoise, and deblur images and videos. It serves as a comprehensive system for image super-resolution and video quality restoration, providing the necessary infrastructure to recover fine visual details and increase pixel density. The project distinguishes itself through specialized toolkits for facial image enhancement and high-fidelity face synthesis, as well as a dedicated video quality restoration suite that utilizes deformable convolutions and generative

Alertmanager is a monitoring notification gateway and routing service that deduplicates, groups, and directs alerts to the correct receivers. It functions as a central manager for Prometheus alerts, using a hierarchical routing tree and label-based matchers to dispatch notifications to external services. The system employs a peer-to-peer mesh network to coordinate multiple instances in a high availability cluster, ensuring continuous alert processing. It features a dedicated inhibition engine and grouping mechanisms to reduce notification noise by suppressing redundant alerts when related iss

Dawarich is a self-hosted location history manager and travel journaling platform. It functions as a personal travel archive that collects GPS coordinates and movement data, providing a private alternative to proprietary tracking services. The system utilizes a PostgreSQL geospatial database to store coordinates, visits, and custom geofence boundaries. The project distinguishes itself as a geospatial data converter and visualization tool, capable of transforming location history between formats such as GPX, KML, and GeoJSON. It allows users to organize GPS tracks and geotagged photos into nam

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

dupeguru is a content-based file deduplicator and cross-platform disk cleanup tool. It functions as a local file management utility designed to identify and remove redundant files to recover disk space. The application identifies identical files across a file system by using content hashing and metadata comparison. This allows it to detect duplicates regardless of their filenames or directory locations. The software covers a range of data management capabilities, including directory scanning, content-based data deduplication, and the consolidation of redundant copies into single versions. It

MMPose is a PyTorch-based pose estimation toolbox and deep learning training pipeline designed for detecting 2D and 3D keypoints on humans, animals, and faces. It serves as a computer vision model zoo and a framework for both 2D pose estimation and 3D pose lifting. The project is distinguished by its modular architecture and extensibility, employing a registry-based system and hierarchical configurations to allow for custom algorithm integration and model pipeline customization. It supports diverse estimation paradigms, including top-down, bottom-up, and two-stage pose lifting workflows. The

Suricata is an open-source network intrusion detection and prevention engine that analyzes live network traffic in real-time to identify and alert on malicious activity. It operates as a rule-based threat detection system, matching traffic against user-defined signatures to detect known attack patterns and policy violations, and can be placed inline to actively block malicious packets before they reach their target. The engine inspects a wide range of application-layer protocols including HTTP, DNS, TLS, SMB, and MQTT, and supports high-performance packet capture through specialized hardware a

Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro

MemOS is an open-source persistent memory layer for AI agents and large language models, providing a self-hosted server that stores and retrieves structured memory across sessions. It enables AI systems to recall user preferences, history, and context without retraining, using a graph-based API and a web management interface for viewing, editing, and organizing memory items, skills, traces, and knowledge bases. The system distinguishes itself through a portable memory interchange protocol that allows memory to be transferred between different AI models, devices, and applications, along with a

Flashlight es una biblioteca de aprendizaje automático en C++ y un framework de aprendizaje profundo diseñado para construir y entrenar redes neuronales. Funciona como una biblioteca de manipulación de tensores y un motor de diferenciación automática que rastrea operaciones para calcular gradientes mediante retropropagación (backpropagation) para la optimización de modelos. El proyecto se distingue por su rol como framework de entrenamiento distribuido, utilizando sincronización de gradientes all-reduce y entornos distribuidos para escalar cargas de trabajo de aprendizaje automático a través de múltiples nodos y dispositivos. Cuenta con una interfaz de memoria agnóstica al backend y gestión basada en RAII para desacoplar las operaciones de tensores del hardware físico. El framework cubre una amplia superficie de capacidades, incluyendo la construcción de arquitecturas de redes neuronales con capas convolucionales, lineales y recurrentes. Proporciona utilidades extensas para álgebra de tensores, gestión y batching de datasets, serialización binaria versionada para estados de modelos y herramientas de monitorización para rastrear métricas de entrenamiento y uso de memoria.

SAHI es un framework de inferencia segmentada y pipeline de visión artificial diseñado para detectar objetos pequeños en imágenes de alta resolución. Proporciona un sistema para dividir imágenes grandes en parches superpuestos para evitar la pérdida de detalle que ocurre típicamente durante la reducción de escala estándar del modelo, junto con una utilidad de mosaico de imágenes y un kit de herramientas de conjunto de datos COCO. El proyecto se distingue por ofrecer un wrapper de predicción agnóstico al modelo que estandariza diferentes frameworks de aprendizaje automático en una interfaz unificada. Esto le permite implementar inferencia segmentada y detección de objetos a través de varios backends de modelos mientras mantiene un formato de salida consistente. Más allá de la inferencia, el framework cubre la gestión de conjuntos de datos para formatos COCO y YOLO, incluyendo herramientas para el corte de imágenes anotadas, remapeo de categorías y fusión de conjuntos de datos. También incluye una suite para la evaluación y monitoreo del rendimiento del modelo, con cálculo de métricas para precisión y recall, análisis de errores de detección y visualización de resultados. El conjunto de herramientas es accesible a través de una interfaz de línea de comandos para automatizar flujos de trabajo de inferencia a través de directorios de imágenes y flujos de video.

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

mmaction2 es un kit de herramientas de comprensión de video de PyTorch diseñado para entrenar y evaluar modelos de aprendizaje profundo. Sirve como un framework para el reconocimiento de acciones, localización temporal y detección de acciones espaciotemporales, proporcionando herramientas especializadas tanto para el análisis de video basado en píxeles como para el reconocimiento de acciones basado en esqueletos. El proyecto se distingue por una arquitectura modular que cuenta con descubrimiento de componentes basado en registro y ensamblaje de modelos jerárquico impulsado por configuración. Admite la fusión de características multimodales, integrando marcos RGB, flujo óptico y audio, e incluye capacidades para la recuperación de clips de video mediante texto y predicción de video zero-shot. A grandes rasgos, el framework cubre la ingeniería de conjuntos de datos de video, incluyendo la estandarización de anotaciones y el muestreo de fotogramas, así como el entrenamiento y evaluación integral de modelos. Proporciona utilidades para el entrenamiento distribuido, destilación de conocimientos y optimización de inferencia mediante reparametrización de modelos. La base de código admite la exportación de modelos ONNX y la contenerización del entorno para el despliegue a través de diferentes nodos de cómputo.