13 个仓库
Tools for organizing distributed processes into logical units for isolated communication and data exchange.
Distinct from Group Messaging Systems: Distinct from general group management: focuses on distributed process topology and communication isolation rather than user permissions.
Explore 13 awesome GitHub repositories matching devops & infrastructure · Process Grouping Utilities. Refine with filters or upvote what's useful.
This project is a Node.js process manager, runtime environment, and production deployment orchestrator. It provides the foundational system components required to run, monitor, and restart applications in the background to ensure continuous service availability. The system distinguishes itself through a built-in load balancer that distributes network traffic across multiple process instances to utilize all available CPU cores. It includes a real-time process monitor with a terminal-based dashboard for tracking server health, CPU and memory usage, and aggregated logs. The tool covers a broad
Organizes applications into namespaces to perform bulk operations like restarting or filtering logs.
This project is a comprehensive educational resource and technical documentation suite for learning and developing deep learning models. It serves as an open-source textbook, implementation manual, and framework tutorial designed to guide users through the mathematical foundations and practical application of neural networks. The resource provides detailed instructional content on building various model architectures, including convolutional and recurrent neural networks. It includes a dedicated distributed training guide and a learning path that covers the fundamentals of tensors, automatic
Provides instructions on coordinating distributed training processes to prevent race conditions.
Accelerate is a PyTorch distributed training library that abstracts the boilerplate required to run models across multiple GPUs, TPUs, and CPUs. It functions as a deep learning model scaler and distributed hardware orchestrator, allowing the same training script to run on different hardware backends without modifying the core logic. The project provides a distributed training command line interface for configuring compute environments and launching jobs across single or multi-node clusters. It includes a mixed precision training framework to implement FP16 and BF16 precision, reducing memory
Provides barrier mechanisms specifically for synchronizing distributed machine learning training processes.
BasicSR is a PyTorch-based image restoration toolbox and framework designed for training and deploying deep learning models to upscale, denoise, and deblur images and videos. It serves as a comprehensive system for image super-resolution and video quality restoration, providing the necessary infrastructure to recover fine visual details and increase pixel density. The project distinguishes itself through specialized toolkits for facial image enhancement and high-fidelity face synthesis, as well as a dedicated video quality restoration suite that utilizes deformable convolutions and generative
Implements execution guards to ensure specific tasks run only on the master process during distributed training.
Initializes distributed process groups with communication backends and unique ranks for training.
sshcode 是一个远程 IDE 部署工具,通过 SSH 在远程服务器上安装并运行 VS Code 编辑器,以实现基于浏览器的访问。它作为一个 SSH 远程 IDE 部署器和隧道代理,通过安全外壳加密隧道将远程编辑器实例路由到本地浏览器端口。 该系统将本地编辑器设置和扩展同步到远程服务器,以确保开发环境的一致性。这包括在本地机器和远程文件系统之间备份和传输用户配置。 该项目管理远程开发的完整生命周期,包括服务器二进制文件的分发、远程进程编排以及基于文件系统的设置同步。它通过管理分布式机器之间的连接和环境一致性,实现了远程文件系统代码编辑。
Installs and manages the lifecycle of a code editor instance running on a remote server via SSH commands.
This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr
Organizes processes into custom communication units to enable flexible, isolated data exchange strategies.
该项目是一个综合性教育计划和深度学习框架,旨在通过 Notebook 和代码示例教授 PyTorch 深度学习实践。它作为一个用于构建、训练和部署神经网络的高级库,充当模型训练编排器,协调 PyTorch 模型、优化器和损失函数。 该项目为计算机视觉、自然语言处理和表格数据预处理提供了专门的工具包。它通过高级训练控制脱颖而出,例如判别式学习率、用于自定义训练逻辑的双向回调系统,以及自动化设备放置和训练循环的高级学习器抽象。 该框架涵盖了广泛的能力面,包括自动化数据流水线构建、模型架构分析以及跨分类、回归和分割任务的性能评估。它还包括用于跨多个 GPU 进行分布式训练的工具、用于内存优化的混合精度训练,以及对医学影像数据的专门支持。 该项目以一系列 Jupyter Notebook 的形式交付。
fastai implements a barrier that forces all sub-processes in a distributed group to synchronize before proceeding.
该项目是一个 Go 进程监督器,旨在启动和监控多个后台程序,并具有自动重启和生命周期管理功能。它作为一个用于协调守护进程执行的系统,通过集中配置确保连续运行。 该监督器的特色在于多个远程管理接口,包括进程管理 REST API、XML-RPC 控制器以及用于监控和控制进程的内置 Web 仪表板。它具有一个 Prometheus 监控导出器,通过专用 HTTP 端点提供实时性能指标,并使用基本身份验证保护这些远程接口。 该系统涵盖了广泛的功能领域,包括具有基于大小轮转的日志管理、通过 TCP 和 HTTP 轮询的自动健康检查,以及通过基于组的优先级调度进行的服务生命周期编排。它还提供事件驱动的操作触发器、用于清理僵尸进程的子进程收割器,以及用于手动进程控制的命令行界面。 该软件包括用于配置模板生成的实用程序,并可以作为系统服务使用自定义环境文件进行集成。
Coordinates the specific order of process startup and shutdown using priority-based group scheduling.
Lightly is a self-supervised learning framework and computer vision data curation tool designed to manage large image datasets and train models on unlabeled data. It functions as a PyTorch vision library and dataset management SDK, providing tools to convert raw images into high-dimensional vectors for similarity search, visualization, and feature extraction. The project implements a variety of self-supervised architectures, including MoCo, SimCLR, VICReg, Barlow Twins, and masked image modeling. It distinguishes itself by combining these learning frameworks with active learning capabilities,
Coordinates synchronization between training processes and restricts specific functions to the primary rank.
This is a structured deep learning curriculum for programmers, delivered as a collection of Jupyter notebooks. It teaches the fundamentals of training neural networks for computer vision, natural language processing, tabular data analysis, and collaborative filtering using PyTorch and the fastai library. The course is designed to be hands-on, guiding learners from building a training loop from scratch to fine-tuning pretrained models for a variety of practical tasks. The curriculum distinguishes itself by covering the full lifecycle of a deep learning project, from data preparation and augmen
Places a barrier that waits for all processes in the PyTorch process group before proceeding.
Organizes distributed processes into logical units for isolated communication and data exchange.
RLinf is a distributed reinforcement learning orchestrator and embodied AI training framework. It provides the infrastructure to train vision-language-action models and robotic policies using a combination of reinforcement learning and supervised fine-tuning. The system is designed for scaling workloads across GPU clusters, managing the placement of actors, rollout workers, and environment components. It features a specialized robotics data collection pipeline for gathering teleoperated demonstrations and simulation trajectories into standardized replay buffers, alongside a hardware interface
Launches and commands groups of identical worker processes to execute functions simultaneously across a distributed system.