NVIDIA/nccl

Features

• Distributed GPU Training - Synchronizing data and model gradients across multiple GPUs in a cluster to accelerate machine learning training.
• Device-Initiated Communication - NCCL executes data exchange operations directly from device code rather than relying on host-side orchestration.
• Collective GPU Communication - A set of optimized primitives for collective and point-to-point data exchange across multiple GPUs in single or multi-node systems.
• CUDA Libraries - A high-performance library for executing all-reduce, broadcast, and all-gather operations using CUDA and NVIDIA GPU interconnects.
• Communication-Computation Overlap - Implements techniques to hide network latency by overlapping gradient synchronization with active GPU computation.
• Rank-Based Broadcasting - Copies data from a single root rank to the receive buffers of all other participating ranks.
• Distributed GPU Computing - A low-level communication layer that synchronizes data and manages device communicators for large-scale distributed training and inference.
• Communication Kernels - NCCL implements specialized data patterns by fusing reduction and broadcast operations into computation kernels.
• Communicator Initialization - NCCL creates a device-side communication object based on host-side requirements to manage data exchange.
• Reduce-Scatter Operations - NCCL distributes a reduced set of data across all GPUs where each device receives a distinct portion.
• ML Library Integrations - Provides the necessary interfaces to connect high-performance communication primitives with deep learning frameworks and tensor libraries.
• Multi-Node - Coordinates GPU communication across multiple physical nodes by establishing NVLink domains and identifiers.
• Collective Communication Operations - NCCL synchronizes data across multiple GPUs using collective operations to ensure all participants reach a consistent state.
• Distributed - NCCL performs a reduction across multiple sources and copies the resulting value to destinations in a single operation.
• RDMA GPU Transfers - Utilizes RDMA and high-speed interconnects to move data directly between GPU memories across different physical nodes.
• InfiniBand NCCL Configurations - Optimizes RDMA communication by configuring InfiniBand interfaces, timeouts, and retry counts.
• Domain Scatter Operations - NCCL distributes contiguous segments of a buffer from a root rank to all participating ranks.
• All-to-All Collective Communication - Performs many-to-many exchanges where every device sends and receives distinct data blocks to and from every other device.
• Device-to-Device Reductions - NCCL combines data arrays from all devices using a specified operation and stores the result on a root or all devices.
• Broadcast Collectives - NCCL copies data from a single root device to all other participating devices in a group.
• Gather Collective Operations - NCCL collects data from all ranks and concatenates the result into a buffer on root or all ranks.
• GPU Data Broadcasting - Copies data from a local source buffer to symmetric destination buffers across multiple peer GPUs.
• GPU Networking - Managing high-speed data movement between GPUs across different physical nodes using InfiniBand, NVLink, or TCP/IP.
• GPU Peer-to-Peer Memory Access - Manages direct peer-to-peer memory access between GPUs using distance cutoffs and buffer access controls.
• Signal-Based Synchronization - Uses lightweight atomic memory flags as notification signals between peer ranks to coordinate execution flow.
• Point-to-Point Communication Primitives - NCCL transfers data directly between two GPUs using two-sided or one-sided remote memory access.
• Scatter Collective Operations - NCCL distributes chunks of a large data buffer from a root device across all participating devices.
• Scatter-Gather Patterns - NCCL collects data buffers from all participating devices and aggregates them into a single buffer on the root device.
• Device-Side Operation Launch - Allows communication primitives to be executed directly from GPU kernels, eliminating CPU overhead from the critical path.
• GPU Memory Orchestration - A system for registering memory windows and allocating aligned device buffers to maximize bandwidth during peer-to-peer transfers.
• Accelerator Memory Registration - NCCL creates a registered memory window to enable one-sided remote memory access and peer-to-peer mapping.
• Communicator-Based Process Groupings - Synchronizes communication groups across multiple operating system processes using shared memory and network handshakes.
• Remote Direct Memory Access - NCCL schedules a device-initiated, one-sided data transfer from a local buffer to a remote buffer.
• Remote GPU Memory Access - NCCL reads or writes data directly to a remote registered memory window without requiring the target process's active participation.
• Distributed - NCCL executes element-wise mathematical operations such as sum, product, or average across multiple GPUs.
• Group Initialization - NCCL creates a coordination group of GPUs across nodes using unique identifiers and configuration attributes.
• Communicator Group Management - NCCL creates, grows, and shrinks communicators to define the set of GPUs participating in data exchanges.
• GPU Execution Coordination - Coordinating execution flow across multiple devices using network barriers and atomic signals to ensure memory consistency.
• Cross-Device Memory Coordination - NCCL coordinates load/store accessible memory and barriers to synchronize data across multiple devices.
• Graph-Capture Buffer Registrations - Implements GPU kernels that register shared IPC buffers to enable fast multi-GPU communication within CUDA graphs.
• Fused Collective Fusions - Combines multiple collective communication primitives into single GPU kernels to increase throughput and reduce launch overhead.
• Concurrent Communicator Management - NCCL executes operations across multiple communicators per device using implicit or explicit ordering to prevent deadlocks.
• Collective Group Tuning - Configures execution behavior, network module selection, and kernel resource allocation for collective groups.
• Communication Resource Tuning - Adjusts the number of blocks, threads, and buffers used for communication to balance throughput and memory.
• Asynchronous Collective Execution - Enqueues collective operations to a device stream to allow computation to run in the background.
• Custom Reduction Operators - NCCL defines specialized logic for combining data during collective operations to support non-standard mathematical transformations.
• In-Place Operations - NCCL executes data exchange functions where input and output buffers are the same to reduce memory overhead.
• Collective Workflow Capture - Records collective and peer-to-peer operations into a graph to reduce CPU overhead by launching tasks as a single unit.
• Hardware Buffer Zero-Copy - Maps user memory regions directly to network interfaces to eliminate intermediate CPU copies during data transfers.
• GPU Stream Synchronization - Coordinates multiple GPU device streams to ensure they all reach a global synchronization point.
• Multi-Device Process Mapping - NCCL associates multiple GPU ranks with a single operating system process to optimize resource utilization.
• Registration & Windows - Configuring registered memory windows and zero-copy buffers to enable direct remote memory access between devices.
• Data Throughput Optimizers - Adjusts chunk sizes and network path diversity to maximize data throughput across hardware topologies.
• Barrier Synchronization - Executes network barriers across a group of peers to ensure all participating threads reach a common point.
• Multicast Group Communication - NCCL reduces data across devices using hardware-level multicast features on supported datacenter GPUs.
• Communicator Behavioral Settings - Configures execution parameters such as blocking mode to optimize the efficiency of GPU data exchanges.
• Network Traffic Prioritization - Prioritizes network bandwidth and manages Quality of Service by mapping communicators to specific traffic classes.
• Network Interface Mapping - Controls the mapping of network interfaces to specific GPUs to optimize multi-node data movement.
• Direct Rank-to-Rank Transfers - NCCL transfers data directly between two specific GPU ranks to implement custom communication patterns.
• One-Sided Network Transfers - NCCL sends signals and transfers data across the network without requiring immediate receiver participation.
• Direct Memory Load-Store Exchanges - NCCL transfers data between GPUs using memory load and store operations over direct peer-to-peer connections.
• Ring-Allreduce Algorithms - Implements circular communication chains to optimize bandwidth for reductions and broadcasts across a cluster of GPUs.
• Socket Option Configurations - Manages TCP/IP communication between GPUs by configuring low-level socket parameters and retry logic.
• Topology-Aware Routing - Optimizes data movement by selecting the fastest available interconnects based on the physical layout of GPUs and nodes.
• Communication Channels - NCCL establishes multiple independent communication channels on a single GPU using blocking or non-blocking initialization.
• CPU Affinity Binding - Binds internal threads to specific processor cores based on hardware proximity to minimize latency.
• Device Group Synchronizations - Coordinates execution across different cooperative groups and rank subsets using memory or network barriers.
• Direct-Pointer Memory Access - Implements direct access to raw memory addresses for high-performance GPU kernel communication.
• Device-Specific Memory Allocators - NCCL manages the allocation of device memory specifically optimized for communication primitives.
• VMM-Compatible Allocators - Provides specialized memory allocation that meets specific alignment and registration requirements for network RDMA.
• Cross-Memory Transfer Utilities - NCCL transfers data using various memory pointers to enable communication between GPUs across local, host, and unified memory.
• Communicator Partitioning - NCCL splits an existing group into smaller sub-groups based on a color value or removes specific ranks.
• Communicator Splitting - NCCL divides an existing communicator into multiple new groups based on assigned color values and keys.
• Multimem Buffer Synchronization - NCCL moves data between logical buffers that map across all ranks using hardware multicast paths.
• GPU Communication Bindings - Exposes core high-performance communication primitives through a Python interface for easier integration into ML workflows.
• Asynchronous Operation Trackers - Tracks the progress and completion status of concurrent asynchronous communication tasks.
• GPU Device Property Queries - NCCL retrieves configuration and capability metadata to verify hardware support before allocating resources.
• Cluster Thread Block Synchronizations - Blocks execution until all participating threads across multiple GPUs in a cluster reach a specific barrier session.
• CUDA-Aware MPI Libraries - Integrates with CUDA-aware MPI libraries to create GPU communication groups based on existing MPI communicators.
• In-Network Collective Offloads - Offloads collective communication operations to InfiniBand switch hardware to reduce processor load and data movement.
• Asynchronous Error Handling - Handles network or process failures within asynchronous communication streams to facilitate system recovery.
• Multicast Memory Pointers - NCCL provides a pointer to a multicast-enabled memory region for hardware-accelerated data distribution across devices.
• Non-Contiguous Memory Addressing - NCCL uses device-side lambdas to specify non-contiguous addressing for source and destination buffers during communication.
• Non-Blocking Execution - Implements non-blocking thread behavior to allow multiple threads to manage data exchanges independently without freezing execution.
• Dynamic Group Expansion - NCCL adds new ranks to an existing communication group using a unique identifier to grow the collective.
• Runtime Parameter Tuning - Tunes execution behavior and performance using system settings provided via environment variables or config files.
• Communication Buffer Registrations - NCCL registers a user buffer with the communicator to enable zero-copy optimized data transfers.
• Communication-Compute Balancing - Allocates thread arrays to balance the trade-off between communication performance and available compute resources.
• Concurrency and Parallelism - Optimized primitives for multi-GPU communication.

Star history