Awesome GitHub RepositoriesData Deduplication Tools

This project is a human resources management system built using Spring Boot and Vue. It serves as a platform for managing employee records, professional titles, and organizational hierarchies. The system features a role-based access control framework that maps users to specific roles and resources to secure API endpoints and user interface elements. It includes a real-time communication hub utilizing WebSockets for internal corporate chat and system notifications, as well as a dedicated manager for defining and modifying nested organizational department structures. Additional capabilities co

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

Memori is an AI agent memory middleware platform designed to provide persistent, context-aware recall for language models. It functions as a non-intrusive layer that intercepts outbound model requests to automatically capture interaction history and execution traces, ensuring that agents maintain continuity across sessions without requiring modifications to existing application logic. The platform distinguishes itself through a dual-model storage architecture that maintains information as both structured relational primitives for precise fact retrieval and rolling narrative summaries for situ

This repository is a comprehensive collection of instructional guides and practical examples for Python development, focusing on machine learning, data science, and web scraping. It provides implementations for neural networks, reinforcement learning algorithms, and deep learning architectures using PyTorch, alongside detailed manuals for scientific computing and data visualization. The project distinguishes itself by offering specialized tutorials on concurrent programming to optimize CPU performance and guides for setting up Linux development environments. It covers the implementation of ad

Toastr is a JavaScript toast notification library and client-side alert manager used to display non-blocking alert messages and system notifications in web browsers. It serves as a frontend notification UI for rendering timed messages that provide real-time visual feedback within web applications. The library includes a notification system that supports right-to-left layout mirroring for internationalized applications. It also features a mechanism to prevent duplicate notifications by matching new content against active alerts to avoid redundant stacking. The system manages notification life

Keep is an open-source AIOps alert management platform that aggregates, deduplicates, and orchestrates the lifecycle of alerts from multiple monitoring tools. It functions as a multi-provider integration hub to centralize the flow of data between observability, ticketing, and communication tools. The platform distinguishes itself through incident workflow automation and AI-powered enrichment. It uses a declarative workflow engine to execute multi-step operational sequences and integrates large language models to summarize event data and correlate technical logs for faster incident resolution.

BasicSR is a PyTorch-based image restoration toolbox and framework designed for training and deploying deep learning models to upscale, denoise, and deblur images and videos. It serves as a comprehensive system for image super-resolution and video quality restoration, providing the necessary infrastructure to recover fine visual details and increase pixel density. The project distinguishes itself through specialized toolkits for facial image enhancement and high-fidelity face synthesis, as well as a dedicated video quality restoration suite that utilizes deformable convolutions and generative

Alertmanager is a monitoring notification gateway and routing service that deduplicates, groups, and directs alerts to the correct receivers. It functions as a central manager for Prometheus alerts, using a hierarchical routing tree and label-based matchers to dispatch notifications to external services. The system employs a peer-to-peer mesh network to coordinate multiple instances in a high availability cluster, ensuring continuous alert processing. It features a dedicated inhibition engine and grouping mechanisms to reduce notification noise by suppressing redundant alerts when related iss

Dawarich is a self-hosted location history manager and travel journaling platform. It functions as a personal travel archive that collects GPS coordinates and movement data, providing a private alternative to proprietary tracking services. The system utilizes a PostgreSQL geospatial database to store coordinates, visits, and custom geofence boundaries. The project distinguishes itself as a geospatial data converter and visualization tool, capable of transforming location history between formats such as GPX, KML, and GeoJSON. It allows users to organize GPS tracks and geotagged photos into nam

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

dupeguru is a content-based file deduplicator and cross-platform disk cleanup tool. It functions as a local file management utility designed to identify and remove redundant files to recover disk space. The application identifies identical files across a file system by using content hashing and metadata comparison. This allows it to detect duplicates regardless of their filenames or directory locations. The software covers a range of data management capabilities, including directory scanning, content-based data deduplication, and the consolidation of redundant copies into single versions. It

MMPose is a PyTorch-based pose estimation toolbox and deep learning training pipeline designed for detecting 2D and 3D keypoints on humans, animals, and faces. It serves as a computer vision model zoo and a framework for both 2D pose estimation and 3D pose lifting. The project is distinguished by its modular architecture and extensibility, employing a registry-based system and hierarchical configurations to allow for custom algorithm integration and model pipeline customization. It supports diverse estimation paradigms, including top-down, bottom-up, and two-stage pose lifting workflows. The

Suricata is an open-source network intrusion detection and prevention engine that analyzes live network traffic in real-time to identify and alert on malicious activity. It operates as a rule-based threat detection system, matching traffic against user-defined signatures to detect known attack patterns and policy violations, and can be placed inline to actively block malicious packets before they reach their target. The engine inspects a wide range of application-layer protocols including HTTP, DNS, TLS, SMB, and MQTT, and supports high-performance packet capture through specialized hardware a

Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro

MemOS is an open-source persistent memory layer for AI agents and large language models, providing a self-hosted server that stores and retrieves structured memory across sessions. It enables AI systems to recall user preferences, history, and context without retraining, using a graph-based API and a web management interface for viewing, editing, and organizing memory items, skills, traces, and knowledge bases. The system distinguishes itself through a portable memory interchange protocol that allows memory to be transferred between different AI models, devices, and applications, along with a

Flashlight ist eine C++-Bibliothek für maschinelles Lernen und ein Deep-Learning-Framework zur Erstellung und zum Training neuronaler Netze. Es fungiert als Tensor-Manipulationsbibliothek und Engine für automatische Differenzierung, die Operationen verfolgt, um Gradienten via Backpropagation für die Modelloptimierung zu berechnen. Das Projekt zeichnet sich durch seine Rolle als Framework für verteiltes Training aus, das All-Reduce-Gradientensynchronisation und verteilte Umgebungen nutzt, um Machine-Learning-Workloads über mehrere Nodes und Geräte hinweg zu skalieren. Es verfügt über eine Backend-agnostische Speicherschnittstelle und RAII-basiertes Management, um Tensor-Operationen von der physischen Hardware zu entkoppeln. Das Framework deckt ein breites Funktionsspektrum ab, einschließlich der Konstruktion neuronaler Netzwerkarchitekturen mit konvolutiven, linearen und rekurrenten Schichten. Es bietet umfangreiche Utilities für Tensor-Algebra, Dataset-Management und Batching, versionierte Binärserialisierung für Modellzustände sowie Überwachungswerkzeuge zur Verfolgung von Trainingsmetriken und Speicherauslastung.

SAHI ist ein Sliced-Inference-Framework und eine Computer-Vision-Pipeline, die entwickelt wurde, um kleine Objekte in hochauflösenden Bildern zu erkennen. Es bietet ein System zur Unterteilung großer Bilder in überlappende Patches, um den Detailverlust zu verhindern, der typischerweise bei der Standard-Modell-Herunterskalierung auftritt, sowie ein Bild-Tiling-Dienstprogramm und ein COCO-Datensatz-Toolkit. Das Projekt zeichnet sich durch einen modellagnostischen Vorhersage-Wrapper aus, der verschiedene Machine-Learning-Frameworks in eine einheitliche Schnittstelle standardisiert. Dies ermöglicht die Implementierung von Sliced Inference und Objekterkennung über verschiedene Modell-Backends hinweg bei gleichzeitiger Beibehaltung eines konsistenten Ausgabeformats. Über die Inferenz hinaus deckt das Framework das Datensatzmanagement für COCO- und YOLO-Formate ab, einschließlich Tools für annotiertes Bild-Slicing, Kategorien-Remapping und Datensatz-Zusammenführung. Es enthält zudem eine Suite zur Bewertung und Überwachung der Modellleistung, mit Metrikberechnung für Präzision und Recall, Erkennungsfehleranalyse und Ergebnisvisualisierung. Das Toolset ist über eine Befehlszeilenschnittstelle zugänglich, um Inferenz-Workflows über Bildverzeichnisse und Videostreams hinweg zu automatisieren.

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

mmaction2 ist eine PyTorch-Toolbox für das Videoverständnis, die für das Training und die Evaluierung von Deep-Learning-Modellen entwickelt wurde. Sie dient als Framework für Aktionserkennung, zeitliche Lokalisierung und räumlich-zeitliche Aktionserkennung und bietet spezialisierte Tools sowohl für pixelbasierte Videoanalyse als auch für skelettbasierte Aktionserkennung. Das Projekt zeichnet sich durch eine modulare Architektur mit registerbasierter Komponentenerkennung und hierarchischem, konfigurationsgesteuertem Modell-Assembly aus. Es unterstützt multimodale Feature-Fusion, integriert RGB-Frames, optischen Fluss und Audio und enthält Funktionen für Text-zu-Video-Clip-Retrieval und Zero-Shot-Videovorhersage. Das Framework deckt breit gefächert das Video-Dataset-Engineering ab, einschließlich Annotationsstandardisierung und Frame-Sampling, sowie umfassendes Modelltraining und -evaluierung. Es bietet Dienstprogramme für verteiltes Training, Knowledge Distillation und Inferenzoptimierung durch Modell-Reparametrisierung. Die Codebasis unterstützt den ONNX-Modell-Export und die Containerisierung der Umgebung für das Deployment über verschiedene Rechenknoten hinweg.