Awesome GitHub RepositoriesData Deduplication Tools

This project is a human resources management system built using Spring Boot and Vue. It serves as a platform for managing employee records, professional titles, and organizational hierarchies. The system features a role-based access control framework that maps users to specific roles and resources to secure API endpoints and user interface elements. It includes a real-time communication hub utilizing WebSockets for internal corporate chat and system notifications, as well as a dedicated manager for defining and modifying nested organizational department structures. Additional capabilities co

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

Memori is an AI agent memory middleware platform designed to provide persistent, context-aware recall for language models. It functions as a non-intrusive layer that intercepts outbound model requests to automatically capture interaction history and execution traces, ensuring that agents maintain continuity across sessions without requiring modifications to existing application logic. The platform distinguishes itself through a dual-model storage architecture that maintains information as both structured relational primitives for precise fact retrieval and rolling narrative summaries for situ

This repository is a comprehensive collection of instructional guides and practical examples for Python development, focusing on machine learning, data science, and web scraping. It provides implementations for neural networks, reinforcement learning algorithms, and deep learning architectures using PyTorch, alongside detailed manuals for scientific computing and data visualization. The project distinguishes itself by offering specialized tutorials on concurrent programming to optimize CPU performance and guides for setting up Linux development environments. It covers the implementation of ad

Toastr is a JavaScript toast notification library and client-side alert manager used to display non-blocking alert messages and system notifications in web browsers. It serves as a frontend notification UI for rendering timed messages that provide real-time visual feedback within web applications. The library includes a notification system that supports right-to-left layout mirroring for internationalized applications. It also features a mechanism to prevent duplicate notifications by matching new content against active alerts to avoid redundant stacking. The system manages notification life

Keep is an open-source AIOps alert management platform that aggregates, deduplicates, and orchestrates the lifecycle of alerts from multiple monitoring tools. It functions as a multi-provider integration hub to centralize the flow of data between observability, ticketing, and communication tools. The platform distinguishes itself through incident workflow automation and AI-powered enrichment. It uses a declarative workflow engine to execute multi-step operational sequences and integrates large language models to summarize event data and correlate technical logs for faster incident resolution.

BasicSR is a PyTorch-based image restoration toolbox and framework designed for training and deploying deep learning models to upscale, denoise, and deblur images and videos. It serves as a comprehensive system for image super-resolution and video quality restoration, providing the necessary infrastructure to recover fine visual details and increase pixel density. The project distinguishes itself through specialized toolkits for facial image enhancement and high-fidelity face synthesis, as well as a dedicated video quality restoration suite that utilizes deformable convolutions and generative

Alertmanager is a monitoring notification gateway and routing service that deduplicates, groups, and directs alerts to the correct receivers. It functions as a central manager for Prometheus alerts, using a hierarchical routing tree and label-based matchers to dispatch notifications to external services. The system employs a peer-to-peer mesh network to coordinate multiple instances in a high availability cluster, ensuring continuous alert processing. It features a dedicated inhibition engine and grouping mechanisms to reduce notification noise by suppressing redundant alerts when related iss

Dawarich is a self-hosted location history manager and travel journaling platform. It functions as a personal travel archive that collects GPS coordinates and movement data, providing a private alternative to proprietary tracking services. The system utilizes a PostgreSQL geospatial database to store coordinates, visits, and custom geofence boundaries. The project distinguishes itself as a geospatial data converter and visualization tool, capable of transforming location history between formats such as GPX, KML, and GeoJSON. It allows users to organize GPS tracks and geotagged photos into nam

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

dupeguru is a content-based file deduplicator and cross-platform disk cleanup tool. It functions as a local file management utility designed to identify and remove redundant files to recover disk space. The application identifies identical files across a file system by using content hashing and metadata comparison. This allows it to detect duplicates regardless of their filenames or directory locations. The software covers a range of data management capabilities, including directory scanning, content-based data deduplication, and the consolidation of redundant copies into single versions. It

MMPose is a PyTorch-based pose estimation toolbox and deep learning training pipeline designed for detecting 2D and 3D keypoints on humans, animals, and faces. It serves as a computer vision model zoo and a framework for both 2D pose estimation and 3D pose lifting. The project is distinguished by its modular architecture and extensibility, employing a registry-based system and hierarchical configurations to allow for custom algorithm integration and model pipeline customization. It supports diverse estimation paradigms, including top-down, bottom-up, and two-stage pose lifting workflows. The

Suricata is an open-source network intrusion detection and prevention engine that analyzes live network traffic in real-time to identify and alert on malicious activity. It operates as a rule-based threat detection system, matching traffic against user-defined signatures to detect known attack patterns and policy violations, and can be placed inline to actively block malicious packets before they reach their target. The engine inspects a wide range of application-layer protocols including HTTP, DNS, TLS, SMB, and MQTT, and supports high-performance packet capture through specialized hardware a

Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro

MemOS is an open-source persistent memory layer for AI agents and large language models, providing a self-hosted server that stores and retrieves structured memory across sessions. It enables AI systems to recall user preferences, history, and context without retraining, using a graph-based API and a web management interface for viewing, editing, and organizing memory items, skills, traces, and knowledge bases. The system distinguishes itself through a portable memory interchange protocol that allows memory to be transferred between different AI models, devices, and applications, along with a

Flashlight هي مكتبة تعلم آلي بلغة C++ وإطار عمل للتعلم العميق مصمم لبناء وتدريب الشبكات العصبية. تعمل كمكتبة لمعالجة الموترات (Tensors) ومحرك للتمايز التلقائي يتتبع العمليات لحساب التدرجات عبر الانتشار العكسي (Backpropagation) لتحسين النموذج. يتميز المشروع بدوره كإطار عمل للتدريب الموزع، حيث يستخدم مزامنة التدرج (All-reduce) والبيئات الموزعة لتوسيع نطاق أحمال عمل التعلم الآلي عبر عقد وأجهزة متعددة. يتميز بواجهة ذاكرة غير مرتبطة بالخلفية وإدارة تعتمد على RAII لفصل عمليات الموتر عن الأجهزة الفعلية. يغطي إطار العمل مساحة قدرة واسعة بما في ذلك بناء بنيات الشبكات العصبية مع طبقات تلافيفية وخطية ومتكررة. يوفر أدوات واسعة النطاق لجبر الموترات، وإدارة مجموعات البيانات وتجميعها، وتسلسل ثنائي مرقم لحالات النموذج، وأدوات مراقبة لتتبع مقاييس التدريب واستخدام الذاكرة.

SAHI هو إطار عمل للاستدلال المقطع (sliced inference) وخط معالجة رؤية الكمبيوتر مصمم لاكتشاف الأشياء الصغيرة في الصور عالية الدقة. يوفر نظاماً لتقسيم الصور الكبيرة إلى رقع متداخلة لمنع فقدان التفاصيل الذي يحدث عادةً أثناء تقليل حجم النموذج القياسي، إلى جانب أداة تبليط الصور ومجموعة أدوات بيانات COCO. يتميز المشروع بتقديم غلاف تنبؤ محايد للنموذج يوحد أطر عمل تعلم الآلة المختلفة في واجهة موحدة. يسمح هذا بتنفيذ الاستدلال المقطع واكتشاف الأشياء عبر خلفيات نماذج مختلفة مع الحفاظ على تنسيق مخرجات متسق. بعيداً عن الاستدلال، يغطي إطار العمل إدارة مجموعات البيانات لتنسيقات COCO و YOLO، بما في ذلك أدوات لتقطيع الصور المشروحة، وإعادة تعيين الفئات، ودمج مجموعات البيانات. كما يتضمن مجموعة لتقييم ومراقبة أداء النموذج، تتميز بحساب مقاييس الدقة والاستدعاء، وتحليل خطأ الاكتشاف، وتصور النتائج. مجموعة الأدوات متاحة عبر واجهة سطر الأوامر لأتمتة سير عمل الاستدلال عبر أدلة الصور وتدفقات الفيديو.

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

mmaction2 هو صندوق أدوات لفهم الفيديو لـ PyTorch مصمم لتدريب وتقييم نماذج التعلم العميق. يعمل كإطار عمل للتعرف على الإجراءات، والتوطين الزمني، واكتشاف الإجراءات الزمانية المكانية، ويوفر أدوات متخصصة لكل من تحليل الفيديو القائم على البكسل والتعرف على الإجراءات القائم على الهيكل العظمي. يتميز المشروع ببنية معيارية تتميز باكتشاف المكونات القائم على السجل وتجميع النماذج الهرمي القائم على التكوين. يدعم دمج الميزات متعدد الوسائط، ودمج إطارات RGB، والتدفق البصري، والصوت، ويتضمن إمكانيات لاسترجاع مقاطع الفيديو من النص وتنبؤ الفيديو بدون تدريب مسبق (Zero-shot). بشكل عام، يغطي إطار العمل هندسة مجموعات بيانات الفيديو، بما في ذلك توحيد التعليقات التوضيحية وأخذ عينات الإطارات، بالإضافة إلى تدريب وتقييم النماذج الشامل. يوفر أدوات للتدريب الموزع، وتقطير المعرفة، وتحسين الاستنتاج عبر إعادة معلمات النموذج. يدعم الكود المصدري تصدير نموذج ONNX وحاويات البيئة للنشر عبر عقد حوسبة مختلفة.