Awesome GitHub RepositoriesMulti-node Orchestration

Minikube is a command-line tool designed for local Kubernetes development, enabling users to provision and manage full-featured container clusters directly on a workstation. It serves as a local orchestrator that automates the lifecycle of isolated environments, allowing developers to start, stop, pause, and delete clusters to support testing and integration workflows. The project distinguishes itself through its flexible architecture, which supports multiple virtualization drivers and container runtimes to accommodate diverse host environments. It provides deep integration between the host a

Excelize is a library for reading and writing spreadsheet files in the Office Open XML format. It provides a comprehensive suite of tools for programmatically creating, modifying, and analyzing workbooks, worksheets, and cell data, ensuring compatibility across various office software suites through structured XML serialization. The library distinguishes itself with a built-in formula calculation engine that evaluates complex mathematical and logical expressions directly against workbook data. It also features a memory-mapped streaming architecture, which allows for the efficient processing o

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Salt is an infrastructure configuration management tool and orchestration framework designed for large-scale system administration. It functions as a remote execution engine that enables administrators to manage, provision, and enforce declarative states across distributed fleets of servers from a central control point. By utilizing a high-performance message bus, the platform allows for the simultaneous execution of administrative tasks and the maintenance of consistent software configurations across thousands of nodes. The system distinguishes itself through a flexible architecture that sup

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

SSPanel-UIM is a centralized proxy service management system used to deploy and administrate a network of proxy servers and nodes across multiple protocols. It functions as a management panel for coordinating network resources and providing a unified interface for service administration. The project incorporates a subscription billing system with integrated payment gateways to handle recurring fees and usage-based billing. It features a subscription management panel that generates and distributes proxy configuration lists in multiple formats for compatibility with various client applications.

Monolith is a distributed recommendation model framework and asynchronous training engine designed to build and train large-scale deep learning architectures. It functions as a distributed model trainer that processes massive datasets across multiple compute nodes using asynchronous update mechanisms. The system features a dedicated embedding table manager that creates unique, feature-isolated tables to prevent representation collisions. It also includes a real-time weight updater to capture immediate changes in user interest and data hotspots through continuous parameter synchronization. Th

This is a PyTorch library and framework for self-supervised vision learning. It provides an implementation of masked autoencoders and vision transformers designed to learn image representations by reconstructing masked image patches from unlabeled data. The project features a distributed training pipeline that scales workloads across multiple GPU nodes. This infrastructure includes multi-node orchestration and gradient accumulation to manage large batch sizes and coordinate resource requests across clusters. The toolkit covers a complete workflow from self-supervised masked pre-training to d

gpt-neox is a distributed training system and framework for building large-scale autoregressive language models. It implements the transformer architecture and provides a toolkit for training models with billions of parameters by distributing weights across compute clusters. The framework distinguishes itself through extensive support for distributed model parallelism, including pipeline and sequence parallelism, to overcome single-device memory limits. It further supports sparse model architectures using a mixture of experts system with Sinkhorn-based routing. The project covers a broad ran

OpenNMT-py is a PyTorch neural machine translation framework used for training and deploying neural machine translation and large language models. It functions as a distributed model training system, an inference engine, and a toolkit for fine-tuning large language models. The framework distinguishes itself with a dedicated toolkit for adapting large language models through low-rank adaptation, quantization, and instruction tuning. It also includes a neural machine translation server that allows trained models to be hosted and exposed via REST API endpoints. The project covers a broad range

Deis is an open-source, self-hosted Platform-as-a-Service that deploys and manages containerized applications on a CoreOS cluster using a Heroku-inspired git push workflow. It accepts application code via git push, automatically builds a Docker image, and runs it as a container on the cluster, with systemd and etcd providing service discovery and configuration management. The platform provides a developer experience modeled after Heroku, with a command-line interface for creating, scaling, configuring, and monitoring applications. It hosts a private Git remote per application that triggers th

Konga is a web-based administration panel for managing Kong API Gateway entities, nodes, and configurations without requiring direct command-line or API calls. It provides a visual interface for configuring routes, services, plugins, and upstreams, and supports connecting to and switching between multiple distributed gateway instances from a single control plane. The panel includes role-based access control that regulates which administrators can view or modify gateway configurations through separate user accounts and permissions. It also offers polling-based health monitoring that periodical

Polyaxon is a Kubernetes-native machine learning orchestration platform and MLOps pipeline orchestrator. It serves as a control plane for managing distributed deep learning workloads, automated machine learning pipelines, and experiment tracking. The platform distinguishes itself through specialized services for distributed training management, including MPI-based coordination for PyTorch and TensorFlow. It provides an automated hyperparameter optimization service utilizing Bayesian, random, and grid search algorithms, alongside managed interactive AI workspaces for launching Jupyter notebook

x-ui-yg is a web-based proxy panel and multi-protocol manager used to deploy and manage network proxy protocols for bypassing internet censorship. It serves as a centralized administrator for secure network tunnels, providing a dashboard to obscure internet traffic and maintain user privacy. The project functions as a proxy subscription server, generating aggregated client subscription links and configuration files locally to remove reliance on third-party conversion tools. It also acts as a CDN proxy orchestrator, allowing the use of CDN domains and encryption to mask server identities and p

هذا المشروع عبارة عن بوابة API ووحدة تحكم دخول (ingress controller) مصممة لإدارة حركة المرور، والأمان، واتصال الخدمة داخل بيئات Kubernetes. يعمل كوحدة تحكم تراقب حالة العنقود لمطابقة تكوينات البوابة مع تعريفات البنية التحتية المطلوبة، مما يضمن بقاء سياسات الشبكة وقواعد التوجيه متسقة عبر عمليات النشر الموزعة. يتميز النظام بخط معالجة طلبات معياري يسمح بحقن منطق مخصص للتعامل مع التحويلات، وفحوصات الأمان، والتسجيل. يدعم النظام إدارة البنية التحتية التصريحية، مما يتيح للمستخدمين تحديد سياسات حركة المرور وإعدادات البوابة من خلال ملفات (manifests) خاضعة للتحكم في الإصدار. ومن خلال الاندماج مع تخزين الأسرار الخارجي وتوفير تحكم مركزي عبر مثيلات بوابة متعددة، يحافظ النظام على نهج موحد لفرض السياسات وإدارة الهوية لمستهلكي API. بعيداً عن التوجيه الأساسي، توفر المنصة قدرات شاملة لإدارة حركة المرور، بما في ذلك موازنة التحميل، ومراقبة الصحة، وتعديل مسار الطلب لبروتوكولات HTTP وTCP وgRPC. يسهل النظام الاتصال الآمن من خلال إدارة الشهادات المتكاملة ويسمح بتجميع سياسات المستهلك لضمان تحكم متسق في الوصول عبر معماريات الخدمة.