Awesome GitHub RepositoriesData Deduplication Tools

This project is a human resources management system built using Spring Boot and Vue. It serves as a platform for managing employee records, professional titles, and organizational hierarchies. The system features a role-based access control framework that maps users to specific roles and resources to secure API endpoints and user interface elements. It includes a real-time communication hub utilizing WebSockets for internal corporate chat and system notifications, as well as a dedicated manager for defining and modifying nested organizational department structures. Additional capabilities co

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

Memori is an AI agent memory middleware platform designed to provide persistent, context-aware recall for language models. It functions as a non-intrusive layer that intercepts outbound model requests to automatically capture interaction history and execution traces, ensuring that agents maintain continuity across sessions without requiring modifications to existing application logic. The platform distinguishes itself through a dual-model storage architecture that maintains information as both structured relational primitives for precise fact retrieval and rolling narrative summaries for situ

This repository is a comprehensive collection of instructional guides and practical examples for Python development, focusing on machine learning, data science, and web scraping. It provides implementations for neural networks, reinforcement learning algorithms, and deep learning architectures using PyTorch, alongside detailed manuals for scientific computing and data visualization. The project distinguishes itself by offering specialized tutorials on concurrent programming to optimize CPU performance and guides for setting up Linux development environments. It covers the implementation of ad

Toastr is a JavaScript toast notification library and client-side alert manager used to display non-blocking alert messages and system notifications in web browsers. It serves as a frontend notification UI for rendering timed messages that provide real-time visual feedback within web applications. The library includes a notification system that supports right-to-left layout mirroring for internationalized applications. It also features a mechanism to prevent duplicate notifications by matching new content against active alerts to avoid redundant stacking. The system manages notification life

Keep is an open-source AIOps alert management platform that aggregates, deduplicates, and orchestrates the lifecycle of alerts from multiple monitoring tools. It functions as a multi-provider integration hub to centralize the flow of data between observability, ticketing, and communication tools. The platform distinguishes itself through incident workflow automation and AI-powered enrichment. It uses a declarative workflow engine to execute multi-step operational sequences and integrates large language models to summarize event data and correlate technical logs for faster incident resolution.

BasicSR is a PyTorch-based image restoration toolbox and framework designed for training and deploying deep learning models to upscale, denoise, and deblur images and videos. It serves as a comprehensive system for image super-resolution and video quality restoration, providing the necessary infrastructure to recover fine visual details and increase pixel density. The project distinguishes itself through specialized toolkits for facial image enhancement and high-fidelity face synthesis, as well as a dedicated video quality restoration suite that utilizes deformable convolutions and generative

Alertmanager is a monitoring notification gateway and routing service that deduplicates, groups, and directs alerts to the correct receivers. It functions as a central manager for Prometheus alerts, using a hierarchical routing tree and label-based matchers to dispatch notifications to external services. The system employs a peer-to-peer mesh network to coordinate multiple instances in a high availability cluster, ensuring continuous alert processing. It features a dedicated inhibition engine and grouping mechanisms to reduce notification noise by suppressing redundant alerts when related iss

Dawarich is a self-hosted location history manager and travel journaling platform. It functions as a personal travel archive that collects GPS coordinates and movement data, providing a private alternative to proprietary tracking services. The system utilizes a PostgreSQL geospatial database to store coordinates, visits, and custom geofence boundaries. The project distinguishes itself as a geospatial data converter and visualization tool, capable of transforming location history between formats such as GPX, KML, and GeoJSON. It allows users to organize GPS tracks and geotagged photos into nam

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

dupeguru is a content-based file deduplicator and cross-platform disk cleanup tool. It functions as a local file management utility designed to identify and remove redundant files to recover disk space. The application identifies identical files across a file system by using content hashing and metadata comparison. This allows it to detect duplicates regardless of their filenames or directory locations. The software covers a range of data management capabilities, including directory scanning, content-based data deduplication, and the consolidation of redundant copies into single versions. It

MMPose is a PyTorch-based pose estimation toolbox and deep learning training pipeline designed for detecting 2D and 3D keypoints on humans, animals, and faces. It serves as a computer vision model zoo and a framework for both 2D pose estimation and 3D pose lifting. The project is distinguished by its modular architecture and extensibility, employing a registry-based system and hierarchical configurations to allow for custom algorithm integration and model pipeline customization. It supports diverse estimation paradigms, including top-down, bottom-up, and two-stage pose lifting workflows. The

Suricata is an open-source network intrusion detection and prevention engine that analyzes live network traffic in real-time to identify and alert on malicious activity. It operates as a rule-based threat detection system, matching traffic against user-defined signatures to detect known attack patterns and policy violations, and can be placed inline to actively block malicious packets before they reach their target. The engine inspects a wide range of application-layer protocols including HTTP, DNS, TLS, SMB, and MQTT, and supports high-performance packet capture through specialized hardware a

Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro

MemOS is an open-source persistent memory layer for AI agents and large language models, providing a self-hosted server that stores and retrieves structured memory across sessions. It enables AI systems to recall user preferences, history, and context without retraining, using a graph-based API and a web management interface for viewing, editing, and organizing memory items, skills, traces, and knowledge bases. The system distinguishes itself through a portable memory interchange protocol that allows memory to be transferred between different AI models, devices, and applications, along with a

Flashlight este o bibliotecă C++ de machine learning și un framework de deep learning conceput pentru construirea și antrenarea rețelelor neuronale. Acesta funcționează ca o bibliotecă de manipulare a tensorilor și un motor de diferențiere automată care urmărește operațiunile pentru a calcula gradienții prin backpropagation pentru optimizarea modelului. Proiectul se distinge prin rolul său de framework de antrenare distribuită, utilizând sincronizarea gradienților all-reduce și medii distribuite pentru a scala workload-urile de machine learning pe mai multe noduri și dispozitive. Dispune de o interfață de memorie agnostică față de backend și gestionare bazată pe RAII pentru a decupla operațiunile cu tensori de hardware-ul fizic. Framework-ul acoperă o suprafață largă de capabilități, inclusiv construcția de arhitecturi de rețele neuronale cu straturi convoluționale, liniare și recurente. Oferă utilitare extinse pentru algebră tensorială, gestionarea și batching-ul seturilor de date, serializare binară versionată pentru stările modelelor și instrumente de monitorizare pentru urmărirea metricilor de antrenare și a utilizării memoriei.

SAHI este un framework de inferență prin tăiere (sliced inference) și un pipeline de computer vision conceput pentru a detecta obiecte mici în imagini de înaltă rezoluție. Acesta oferă un sistem pentru divizarea imaginilor mari în patch-uri suprapuse pentru a preveni pierderea detaliilor care apare de obicei în timpul downscaling-ului standard al modelelor, alături de un utilitar de tiling al imaginilor și un toolkit pentru seturi de date COCO. Proiectul se distinge prin oferirea unui wrapper de predicție model-agnostic care standardizează diferite framework-uri de machine learning într-o interfață unificată. Acest lucru îi permite să implementeze inferența prin tăiere și detectarea obiectelor pe diverse backend-uri de modele, menținând în același timp un format de output consistent. Dincolo de inferență, framework-ul acoperă gestionarea seturilor de date pentru formatele COCO și YOLO, inclusiv instrumente pentru tăierea imaginilor adnotate, remaparea categoriilor și fuziunea seturilor de date. Include, de asemenea, o suită pentru evaluarea și monitorizarea performanței modelelor, având calculul metricilor pentru precizie și recall, analiza erorilor de detecție și vizualizarea rezultatelor. Toolkit-ul este accesibil printr-o interfață în linie de comandă pentru automatizarea fluxurilor de lucru de inferență pe directoare de imagini și fluxuri video.

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

mmaction2 este un set de instrumente PyTorch pentru înțelegerea video, conceput pentru antrenarea și evaluarea modelelor de deep learning. Servește ca un framework pentru recunoașterea acțiunilor, localizarea temporală și detectarea acțiunilor spatio-temporale, oferind instrumente specializate atât pentru analiza video bazată pe pixeli, cât și pentru recunoașterea acțiunilor bazată pe schelet. Proiectul se distinge printr-o arhitectură modulară care dispune de descoperirea componentelor bazată pe registru și asamblarea ierarhică a modelelor bazată pe configurație. Suportă fuziunea caracteristicilor multi-modale, integrând cadre RGB, flux optic și audio, și include capabilități pentru recuperarea clipurilor video din text și predicția video zero-shot. În linii mari, framework-ul acoperă ingineria seturilor de date video, inclusiv standardizarea adnotărilor și eșantionarea cadrelor, precum și antrenarea și evaluarea cuprinzătoare a modelelor. Oferă utilitare pentru antrenarea distribuită, distilarea cunoștințelor și optimizarea inferenței prin reparametrizarea modelului. Codul sursă suportă exportul modelelor ONNX și containerizarea mediului pentru implementarea pe diferite noduri de calcul.