Awesome GitHub RepositoriesParallel Programming Frameworks

DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special

Taichi is a domain-specific programming language embedded in Python designed for high-performance numerical computing and computer graphics. It functions as a parallel compiler that translates high-level mathematical expressions into optimized machine instructions, enabling developers to write compute-intensive algorithms that execute across diverse hardware architectures, including CPUs, GPUs, and specialized accelerators. The project distinguishes itself through a hardware-agnostic execution layer that maps parallel operations to multiple backends such as CUDA, Metal, and Vulkan. By utilizi

Codon is an LLVM-based Python compiler and statically typed implementation that translates source code into optimized machine instructions. It functions as a high-performance numerical backend and a GPU computing framework designed to remove runtime overhead. The project implements a compiled alternative to NumPy, translating array logic directly into machine code. It differentiates itself by generating specialized hardware kernels for graphics processors and utilizing static type inference to enable aggressive machine-code optimization. The system provides capabilities for parallel workload

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

ZLUDA is a middleware and translation engine designed to enable the execution of unmodified proprietary compute binaries on non-native graphics hardware. It functions as a compatibility layer that bridges vendor-specific compute interfaces with open standards, allowing software originally restricted to a single hardware ecosystem to operate on alternative graphics processing units. The project achieves this through a combination of dynamic library interception and runtime instruction translation. By replacing standard system libraries and mapping proprietary compute calls to open standards, t

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

Highway is a portable C++ library and hardware abstraction layer designed for writing single instruction multiple data (SIMD) code. It provides a unified interface that maps data-parallel logic to various CPU instruction sets, enabling the development of high-performance software that runs across different processor architectures without requiring architecture-specific assembly. The project features a dynamic instruction dispatcher that selects the most efficient CPU instruction set at runtime based on detected hardware. It also supports static target specialization and extensible mechanisms

HIP ist eine C++-GPU-Kernelsprache und eine plattformübergreifende Laufzeitumgebung, die für das Schreiben portabler Hochleistungs-Rechenanwendungen entwickelt wurde. Sie bietet eine Programmierschnittstelle, die es ermöglicht, eine einzelne Quellcodebasis sowohl auf AMD- als auch auf NVIDIA-GPU-Architekturen auszuführen. Das Projekt fungiert als Kompatibilitätsschicht, die die Konvertierung und Migration bestehender CUDA-Quellcodes zur Ausführung auf AMD-Hardware ermöglicht. Dies wird durch ein Syntax-Mapping erreicht, das CUDA widerspiegelt, sowie durch einen Source-to-Source-Übersetzungsprozess während der Kompilierung. Das Toolkit deckt die breitere Oberfläche der plattformübergreifenden GPGPU-Entwicklung ab, einschließlich heterogener Rechenoptimierung und der Erstellung portabler Kernels. Es nutzt eine Laufzeitabstraktion, um einheitliche API-Aufrufe auf herstellerspezifische Treiberbibliotheken für Speicher- und Kernel-Management abzubilden.

An Open Source Implementation of the Actor Model in C++

The C++ Standard Library for Parallelism and Concurrency

cuda-oxide is an experimental Rust-to-CUDA compiler that lets you write (SIMT) GPU kernels in safe(ish), idiomatic Rust. It compiles standard Rust code directly to PTX — no DSLs, no foreign language bindings, just Rust.

ISPC is a vectorizing compiler and SIMD parallel programming language that implements a single program multiple data model. It serves as a toolchain for translating C-based code with parallel extensions into optimized machine code for various CPU and GPU architectures using an LLVM backend. The compiler is designed for cross-platform SIMD toolchain support, generating specialized instruction sets for x86 SSE/AVX, ARM NEON, and Intel GPU from a single source. It features a runtime dispatch mechanism that selects the most efficient hardware-specific implementation for the current system during

Kokkos C++ Performance Portability Programming Ecosystem: The Programming Model - Parallel Execution and Memory Abstraction

General purpose GPU compute framework built on Vulkan to support 1000s of cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)

RaftLib is an open-source C++ Library that provides a framework for implementing parallel and concurrent data processing pipelines. It is designed to simplify the development of high-performance data processing applications by abstracting away the complexities of parallelism, concurrency, and…

Copyright 2024 NVIDIA Corporation

Legion is a parallel programming model for distributed, heterogeneous machines.

A header-only C++ library for task concurrency