Awesome GitHub RepositoriesMulti-node Orchestration

Minikube is a command-line tool designed for local Kubernetes development, enabling users to provision and manage full-featured container clusters directly on a workstation. It serves as a local orchestrator that automates the lifecycle of isolated environments, allowing developers to start, stop, pause, and delete clusters to support testing and integration workflows. The project distinguishes itself through its flexible architecture, which supports multiple virtualization drivers and container runtimes to accommodate diverse host environments. It provides deep integration between the host a

Excelize is a library for reading and writing spreadsheet files in the Office Open XML format. It provides a comprehensive suite of tools for programmatically creating, modifying, and analyzing workbooks, worksheets, and cell data, ensuring compatibility across various office software suites through structured XML serialization. The library distinguishes itself with a built-in formula calculation engine that evaluates complex mathematical and logical expressions directly against workbook data. It also features a memory-mapped streaming architecture, which allows for the efficient processing o

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Salt is an infrastructure configuration management tool and orchestration framework designed for large-scale system administration. It functions as a remote execution engine that enables administrators to manage, provision, and enforce declarative states across distributed fleets of servers from a central control point. By utilizing a high-performance message bus, the platform allows for the simultaneous execution of administrative tasks and the maintenance of consistent software configurations across thousands of nodes. The system distinguishes itself through a flexible architecture that sup

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

SSPanel-UIM is a centralized proxy service management system used to deploy and administrate a network of proxy servers and nodes across multiple protocols. It functions as a management panel for coordinating network resources and providing a unified interface for service administration. The project incorporates a subscription billing system with integrated payment gateways to handle recurring fees and usage-based billing. It features a subscription management panel that generates and distributes proxy configuration lists in multiple formats for compatibility with various client applications.

Monolith is a distributed recommendation model framework and asynchronous training engine designed to build and train large-scale deep learning architectures. It functions as a distributed model trainer that processes massive datasets across multiple compute nodes using asynchronous update mechanisms. The system features a dedicated embedding table manager that creates unique, feature-isolated tables to prevent representation collisions. It also includes a real-time weight updater to capture immediate changes in user interest and data hotspots through continuous parameter synchronization. Th

This is a PyTorch library and framework for self-supervised vision learning. It provides an implementation of masked autoencoders and vision transformers designed to learn image representations by reconstructing masked image patches from unlabeled data. The project features a distributed training pipeline that scales workloads across multiple GPU nodes. This infrastructure includes multi-node orchestration and gradient accumulation to manage large batch sizes and coordinate resource requests across clusters. The toolkit covers a complete workflow from self-supervised masked pre-training to d

gpt-neox is a distributed training system and framework for building large-scale autoregressive language models. It implements the transformer architecture and provides a toolkit for training models with billions of parameters by distributing weights across compute clusters. The framework distinguishes itself through extensive support for distributed model parallelism, including pipeline and sequence parallelism, to overcome single-device memory limits. It further supports sparse model architectures using a mixture of experts system with Sinkhorn-based routing. The project covers a broad ran

OpenNMT-py is a PyTorch neural machine translation framework used for training and deploying neural machine translation and large language models. It functions as a distributed model training system, an inference engine, and a toolkit for fine-tuning large language models. The framework distinguishes itself with a dedicated toolkit for adapting large language models through low-rank adaptation, quantization, and instruction tuning. It also includes a neural machine translation server that allows trained models to be hosted and exposed via REST API endpoints. The project covers a broad range

Deis is an open-source, self-hosted Platform-as-a-Service that deploys and manages containerized applications on a CoreOS cluster using a Heroku-inspired git push workflow. It accepts application code via git push, automatically builds a Docker image, and runs it as a container on the cluster, with systemd and etcd providing service discovery and configuration management. The platform provides a developer experience modeled after Heroku, with a command-line interface for creating, scaling, configuring, and monitoring applications. It hosts a private Git remote per application that triggers th

Konga is a web-based administration panel for managing Kong API Gateway entities, nodes, and configurations without requiring direct command-line or API calls. It provides a visual interface for configuring routes, services, plugins, and upstreams, and supports connecting to and switching between multiple distributed gateway instances from a single control plane. The panel includes role-based access control that regulates which administrators can view or modify gateway configurations through separate user accounts and permissions. It also offers polling-based health monitoring that periodical

x-ui-yg is a web-based proxy panel and multi-protocol manager used to deploy and manage network proxy protocols for bypassing internet censorship. It serves as a centralized administrator for secure network tunnels, providing a dashboard to obscure internet traffic and maintain user privacy. The project functions as a proxy subscription server, generating aggregated client subscription links and configuration files locally to remove reliance on third-party conversion tools. It also acts as a CDN proxy orchestrator, allowing the use of CDN domains and encryption to mask server identities and p

Este proyecto es una puerta de enlace API y controlador de entrada diseñado para gestionar el tráfico, la seguridad y la conectividad de servicios dentro de entornos Kubernetes. Opera como un controlador que monitorea el estado del clúster para reconciliar las configuraciones de la puerta de enlace con las definiciones de infraestructura deseadas, asegurando que las políticas de red y las reglas de enrutamiento permanezcan consistentes en los despliegues distribuidos. El sistema se distingue por una canalización de solicitudes modular que permite la inyección de lógica personalizada para manejar transformaciones, comprobaciones de seguridad y registro. Admite la gestión de infraestructura declarativa, permitiendo a los usuarios definir políticas de tráfico y configuraciones de puerta de enlace a través de manifiestos controlados por versiones. Al integrarse con el almacenamiento de secretos externo y proporcionar un control centralizado sobre múltiples instancias de puerta de enlace, mantiene un enfoque unificado para la aplicación de políticas y la gestión de identidades para los consumidores de API. Más allá del enrutamiento central, la plataforma proporciona capacidades integrales de gestión de tráfico, incluyendo equilibrio de carga, monitoreo de salud y modificación de rutas de solicitud para protocolos HTTP, TCP y gRPC. Facilita la comunicación segura a través de la gestión de certificados integrada y permite la agrupación de políticas de consumidor para asegurar un control de acceso consistente en todas las arquitecturas de servicio.