Awesome GitHub RepositoriesParallel Programming Frameworks

DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special

Taichi is a domain-specific programming language embedded in Python designed for high-performance numerical computing and computer graphics. It functions as a parallel compiler that translates high-level mathematical expressions into optimized machine instructions, enabling developers to write compute-intensive algorithms that execute across diverse hardware architectures, including CPUs, GPUs, and specialized accelerators. The project distinguishes itself through a hardware-agnostic execution layer that maps parallel operations to multiple backends such as CUDA, Metal, and Vulkan. By utilizi

Codon is an LLVM-based Python compiler and statically typed implementation that translates source code into optimized machine instructions. It functions as a high-performance numerical backend and a GPU computing framework designed to remove runtime overhead. The project implements a compiled alternative to NumPy, translating array logic directly into machine code. It differentiates itself by generating specialized hardware kernels for graphics processors and utilizing static type inference to enable aggressive machine-code optimization. The system provides capabilities for parallel workload

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

ZLUDA is a middleware and translation engine designed to enable the execution of unmodified proprietary compute binaries on non-native graphics hardware. It functions as a compatibility layer that bridges vendor-specific compute interfaces with open standards, allowing software originally restricted to a single hardware ecosystem to operate on alternative graphics processing units. The project achieves this through a combination of dynamic library interception and runtime instruction translation. By replacing standard system libraries and mapping proprietary compute calls to open standards, t

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

Highway este o bibliotecă C++ portabilă și un strat de abstractizare hardware conceput pentru scrierea codului de tip SIMD (Single Instruction Multiple Data). Oferă o interfață unificată care mapează logica de procesare paralelă a datelor către diverse seturi de instrucțiuni CPU, permițând dezvoltarea de software de înaltă performanță care rulează pe diferite arhitecturi de procesoare fără a necesita assembly specific arhitecturii. Proiectul dispune de un dispecer dinamic de instrucțiuni care selectează cel mai eficient set de instrucțiuni CPU la runtime, pe baza hardware-ului detectat. De asemenea, suportă specializarea statică a țintei și mecanisme extensibile pentru adăugarea de noi ținte hardware sau operații SIMD personalizate. Biblioteca acoperă o gamă largă de operații vectoriale, inclusiv aritmetică pe elemente, reducerea benzilor, shuffling și execuție condiționată mascată. Include o bibliotecă matematică vectorizată, un manager de memorie pentru alocare aliniată și operații de load-store mascate, precum și primitive pentru criptografie accelerată hardware. Sunt oferite instrumente pentru compilarea și validarea automată a instrucțiunilor accelerate hardware pe mai multe arhitecturi de procesoare.

HIP este un limbaj de kernel GPU C++ și un runtime cross-platform conceput pentru scrierea aplicațiilor de calcul de înaltă performanță portabile. Oferă o interfață de programare care permite unui singur codebase sursă să fie executat atât pe arhitecturi GPU AMD, cât și NVIDIA. Proiectul funcționează ca un strat de compatibilitate care permite conversia și migrarea codului sursă CUDA existent pentru a rula pe hardware AMD. Acest lucru este realizat printr-o mapare a sintaxei care oglindește CUDA și un proces de traducere source-to-source în timpul compilării. Toolkit-ul acoperă suprafața mai largă a dezvoltării GPGPU cross-platform, inclusiv optimizarea calculului eterogen și crearea de kernel-uri portabile. Utilizează o abstractizare a runtime-ului pentru a mapa apelurile API unificate către bibliotecile de drivere specifice furnizorului pentru gestionarea memoriei și a kernel-ului.

An Open Source Implementation of the Actor Model in C++

The C++ Standard Library for Parallelism and Concurrency

cuda-oxide is an experimental Rust-to-CUDA compiler that lets you write (SIMT) GPU kernels in safe(ish), idiomatic Rust. It compiles standard Rust code directly to PTX — no DSLs, no foreign language bindings, just Rust.

ISPC is a vectorizing compiler and SIMD parallel programming language that implements a single program multiple data model. It serves as a toolchain for translating C-based code with parallel extensions into optimized machine code for various CPU and GPU architectures using an LLVM backend. The compiler is designed for cross-platform SIMD toolchain support, generating specialized instruction sets for x86 SSE/AVX, ARM NEON, and Intel GPU from a single source. It features a runtime dispatch mechanism that selects the most efficient hardware-specific implementation for the current system during

Kokkos C++ Performance Portability Programming Ecosystem: The Programming Model - Parallel Execution and Memory Abstraction

General purpose GPU compute framework built on Vulkan to support 1000s of cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)

RaftLib is an open-source C++ Library that provides a framework for implementing parallel and concurrent data processing pipelines. It is designed to simplify the development of high-performance data processing applications by abstracting away the complexities of parallelism, concurrency, and…

Copyright 2024 NVIDIA Corporation

Legion is a parallel programming model for distributed, heterogeneous machines.

A header-only C++ library for task concurrency