bytedance/bytepsArchived

Features

• Distributed Training Frameworks - Implements a distributed framework for scaling deep neural network training across multiple GPUs and compute nodes.
• Collective GPU Communication - Provides GPU-accelerated collective communication primitives like all-reduce and broadcast to synchronize data across nodes.
• Distributed Training - Scales the training of large deep neural networks across multiple GPUs and compute nodes.
• Distributed Training Coordination - Coordinates the timing and ordering of data partitioning and gradient updates to optimize training speed.
• Multi-Node Training Scaling - Enables the distribution of large-scale model training across multiple GPUs and hardware nodes.
• Distributed Gradient Synchronization - Implements mechanisms for exchanging model gradients across multiple computing nodes.
• Asynchronous Gradient Aggregation - Implements asynchronous gradient aggregation to eliminate global synchronization barriers and increase training throughput.
• Distributed Machine Learning Integrators - Provides interfaces to connect the communication library with deep learning frameworks for distributed training.
• GPU Cluster Communications - Orchestrates high-bandwidth data transfers across GPUs using collective communication primitives and RDMA.
• RDMA GPU Transfers - Accelerates remote data movement using RDMA and page-aligned memory for low-latency GPU transfers.
• Distributed Script Launchers - Ships scripts to launch training processes across workers and servers while coordinating node roles.
• Distributed Task Orchestration - Provides a command-line utility for bootstrapping and managing the execution of training tasks across a cluster.
• GPU Cluster Job Schedulers - Includes a launcher and scheduler to orchestrate task execution and resource allocation across GPU clusters.
• Distributed Learning Communication - Provides a high-performance communication layer to synchronize model states and gradients across distributed training nodes.
• RDMA Networking - Optimizes network latency using Remote Direct Memory Access for high-speed data movement during model training.
• Remote Direct Memory Access - Utilizes remote direct memory access to transfer data between network cards and minimize CPU overhead.
• Hardware Utilization Optimizations - Optimizes hardware utilization through CPU core affinity and memory alignment to reduce latency.
• Asynchronous Training - Supports asynchronous training to eliminate global synchronization barriers and increase total system throughput.
• Distributed Debugging - Offers diagnostics for distributed training failures through backtrace capture and tensor value sampling.
• Shared Memory IPC - Uses shared memory regions for high-speed data exchange between local worker and server processes.
• Hardware Topology Optimizers - Optimizes data transfer paths and concurrency based on the physical layout of GPUs and hardware switches.
• CPU Affinity Binding - Pins execution threads to specific physical cores and memory nodes to reduce context switching and latency.
• Process Affinity and Core Binding - Pins execution threads to physical CPU cores to reduce context switching and cache misses.
• Inter-Process Communication Optimizations - Uses shared-memory segments to accelerate data exchange between worker and server processes on the same machine.
• Distributed Training Timelines - Provides tools for recording and analyzing communication timelines to diagnose bottlenecks in distributed training.
• MLOps and Infrastructure - Distributed DNN training framework.

Star history