Awesome GitHub RepositoriesParallel Programming Frameworks

DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special

Taichi is a domain-specific programming language embedded in Python designed for high-performance numerical computing and computer graphics. It functions as a parallel compiler that translates high-level mathematical expressions into optimized machine instructions, enabling developers to write compute-intensive algorithms that execute across diverse hardware architectures, including CPUs, GPUs, and specialized accelerators. The project distinguishes itself through a hardware-agnostic execution layer that maps parallel operations to multiple backends such as CUDA, Metal, and Vulkan. By utilizi

Codon is an LLVM-based Python compiler and statically typed implementation that translates source code into optimized machine instructions. It functions as a high-performance numerical backend and a GPU computing framework designed to remove runtime overhead. The project implements a compiled alternative to NumPy, translating array logic directly into machine code. It differentiates itself by generating specialized hardware kernels for graphics processors and utilizing static type inference to enable aggressive machine-code optimization. The system provides capabilities for parallel workload

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

ZLUDA is a middleware and translation engine designed to enable the execution of unmodified proprietary compute binaries on non-native graphics hardware. It functions as a compatibility layer that bridges vendor-specific compute interfaces with open standards, allowing software originally restricted to a single hardware ecosystem to operate on alternative graphics processing units. The project achieves this through a combination of dynamic library interception and runtime instruction translation. By replacing standard system libraries and mapping proprietary compute calls to open standards, t

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

Highway 是一个便携式 C++ 库和硬件抽象层，专为编写单指令多数据（SIMD）代码而设计。它提供了一个统一接口，将数据并行逻辑映射到各种 CPU 指令集，从而能够开发出在不同处理器架构上运行的高性能软件，而无需特定于架构的汇编代码。 该项目具有动态指令调度器，可根据检测到的硬件在运行时选择最高效的 CPU 指令集。它还支持静态目标专用化，以及用于添加新硬件目标或自定义 SIMD 操作的可扩展机制。 该库涵盖了广泛的向量操作，包括元素级算术、通道归约、混洗和掩码条件执行。它包括一个向量化数学库、用于对齐分配和掩码加载/存储操作的内存管理器，以及用于硬件加速加密的原语。 提供了用于跨多种处理器架构自动编译和验证硬件加速指令的工具。

HIP 是一种 C++ GPU 内核语言和跨平台运行时，专为编写可移植的高性能计算应用而设计。它提供了一个编程接口，允许单个源代码库在 AMD 和 NVIDIA GPU 架构上执行。 该项目作为兼容层，实现了现有 CUDA 源代码的转换和迁移，以在 AMD 硬件上运行。这是通过镜像 CUDA 的语法映射和编译过程中的源到源翻译来实现的。 该工具包涵盖了更广泛的跨平台 GPGPU 开发领域，包括异构计算优化和可移植内核的创建。它利用运行时抽象将统一 API 调用映射到特定于供应商的驱动程序库，以进行内存和内核管理。

An Open Source Implementation of the Actor Model in C++

The C++ Standard Library for Parallelism and Concurrency

cuda-oxide is an experimental Rust-to-CUDA compiler that lets you write (SIMT) GPU kernels in safe(ish), idiomatic Rust. It compiles standard Rust code directly to PTX — no DSLs, no foreign language bindings, just Rust.

ISPC is a vectorizing compiler and SIMD parallel programming language that implements a single program multiple data model. It serves as a toolchain for translating C-based code with parallel extensions into optimized machine code for various CPU and GPU architectures using an LLVM backend. The compiler is designed for cross-platform SIMD toolchain support, generating specialized instruction sets for x86 SSE/AVX, ARM NEON, and Intel GPU from a single source. It features a runtime dispatch mechanism that selects the most efficient hardware-specific implementation for the current system during

Kokkos C++ Performance Portability Programming Ecosystem: The Programming Model - Parallel Execution and Memory Abstraction

General purpose GPU compute framework built on Vulkan to support 1000s of cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)

RaftLib is an open-source C++ Library that provides a framework for implementing parallel and concurrent data processing pipelines. It is designed to simplify the development of high-performance data processing applications by abstracting away the complexities of parallelism, concurrency, and…

Copyright 2024 NVIDIA Corporation

Legion is a parallel programming model for distributed, heterogeneous machines.

A header-only C++ library for task concurrency